Predicting head movement for rendering virtual objects in augmented or virtual reality systems

ABSTRACT

One embodiment is directed to a user display device comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame and a projection sub system coupled to the housing frame to determine a location of appearance of a display object in a field of view of the user based at least in part on at least one of a detection of a head movement of the user and a prediction of a head movement of the user, and to project the display object to the user based on the determined location of appearance of the display object.

RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.14/212,961, filed on Mar. 14, 2014, which claims the benefit under 35U.S.C §119 to U.S. provisional patent application Ser. No. 61/801,219,filed on Mar. 15, 2013. The foregoing applications are herebyincorporated by reference into the present application in theirentirety.

FIELD OF THE INVENTION

The present invention generally relates to systems and methodsconfigured to facilitate interactive virtual or augmented realityenvironments for one or more users.

BACKGROUND

A number of display systems can benefit from information regarding thehead pose of a viewer or user (i.e., the orientation and/or location ofuser's head).

For instance, head-worn displays (or helmet-mounted displays, or smartglasses) are at least loosely coupled to a user's head, and thus movewhen the user's head moves. If the user's head motions are detected bythe display system, the data being displayed can be updated to take thechange in head pose into account.

As an example, if a user wearing a head-worn display views a virtualrepresentation of a 3D object on the display and walks around the areawhere the 3D object appears, that 3D object can be re-rendered for eachviewpoint, giving the user the perception that he or she is walkingaround an object that occupies real space. If the head-worn display isused to present multiple objects with a virtual space (for instance, arich virtual world), measurements of head pose can be used to re-renderthe scene to match the user's dynamically changing head location andorientation and provide an increased sense of immersion in the virtualspace.

Especially for display systems that fill a substantial portion of theuser's visual field with virtual elements, it is critical that theaccuracy of head-tracking is high and that the overall system latency isvery low from the first detection of head motion to the updating of thelight that is delivered by the display to the user's visual system. Ifthe latency is high, the system can create a mismatch between the user'svestibular and visual sensory systems, and generate motion sickness orsimulator sickness.

Some head-worn displays enable the concurrent viewing of real andvirtual elements—an approach often described as augmented reality ormixed reality. In one such configuration, often referred to as a “videosee-through” display, a camera captures elements of a real scene, acomputing system superimposes virtual elements onto the captured realscene, and a non-transparent display presents the composite image to theeyes. Another configuration is often referred to as an “opticalsee-through” display, in which the user can see through transparent (orsemi-transparent) elements in the display system to view directly thelight from real objects in the environment. The transparent element,often referred to as a “combiner”, superimposes light from the displayover the user's view of the real world.

In both video and optical see-through displays, detection of head posecan enable the display system to render virtual objects such that theyappear to occupy a space in the real world. As the user's head movesaround in the real world, the virtual objects are re-rendered as afunction of head pose, such that the virtual objects appear to remainstable relative to the real world. In the case of an optical see-throughdisplay, the user's view of the real world has essentially a zerolatency while his or her view of the virtual objects has a latency thatdepends on the head-tracking rate, processing time, rendering time, anddisplay frame rate. If the system latency is high, the apparent locationof virtual objects will appear unstable during rapid head motions.

In addition to head-worn display systems, other display systems canbenefit from accurate and low latency head pose detection. These includehead-tracked display systems in which the display is not worn on theuser's body, but is, e.g., mounted on a wall or other surface. Thehead-tracked display acts like a window onto a scene, and as a usermoves his head relative the “window” the scene is re-rendered to matchthe user's changing viewpoint. Other systems include a head-wornprojection system, in which a head-worn display projects light onto thereal world.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users.

One embodiment is directed to a method of operation in a virtual imagesystem or an augmented reality system, the method comprising, for eachof at least some of a plurality of frames being presented to an enduser, determining a location of appearance of a virtual object in afield of view of the end user relative to an end user frame ofreference, and adjusting a presentation of at least one subsequent framebased at least in part on the determined location of appearance of thevirtual object in the field of view of the end user. The virtual objectmay be newly introduced in the field of view of the end user temporallyrelative to previous frames presented to the end user. The newlyintroduced virtual object may be determined to likely attract anattention of the end user. The virtual object may be in a new positionin the frame relative to a position in at least one previous frame. Or,the virtual object may be in a new position as presented to the end userrelative to a previous position of the virtual object as previouslypresented to the end user.

The method may further comprise selecting the virtual object based oninput indicative of an attention of the end user to the virtual object.The input indicative of the attention of the end user to the virtualobject may be based at least in part on an appearance of the virtualobject in a new position as presented to the end user relative to aposition of the virtual object as previously presented to the end user.Or, the input indicative of the attention of the end user to the virtualobject may be based at least in part on how quickly a position of thevirtual object as presented to the end user changes relative to theposition of the virtual object as previously presented to the end user.

The adjusting of the presentation of the at least one subsequent framemay include presenting the at least one subsequent frame with a centerof the at least one subsequent frame shifted toward the determinedlocation of appearance of the virtual object in the field of view of theend user. Or, the adjusting of the presentation of the at least onesubsequent frame may include presenting the at least one subsequentframe with a center of the at least one subsequent frame shifted to thedetermined location of appearance of the virtual object in the field ofview of the end user.

The method may further comprise predicting an occurrence of a headmovement of the end user based at least in part on the determinedlocation of appearance of the virtual object in the field of view of theend user. The method may further comprise estimating at least one valueindicative of an estimated speed of the predicted head movement of theend user, determining at least one value that at least partiallycompensates for the estimated speed of the predicted head movement ofthe end user, and rendering the at least one subsequent frame based atleast in part on the determined value.

The method may further comprise estimating at least one change in thespeed in the predicted head movement of the end user, wherein the atleast one change in the speed occurs between a start of the predictedhead movement and an end of the predicted head movement, and whereinestimating the at least one value indicative of the estimated speed ofthe predicted head movement includes estimating the at least one valueindicative of the estimated speed that at least partially accommodatesfor the estimated changes in the speed in the predicted head movement ofthe end user.

The estimating of the at least one change in the speed in the predictedhead movement of the end user may include estimating the at least onechange between a first defined time after the start of the predictedhead movement and a second defined time before the end of the predictedhead movement.

The method may further comprise estimating at least one value indicativeof an estimated acceleration of the predicted head movement of the enduser, determining at least one value that at least partially compensatesfor the estimated acceleration of the predicted head movement of the enduser, and rendering the at least one subsequent frame based at least inpart on the determined value.

The method may further comprise receiving information indicative of anidentity of the end user, and retrieving at least one user specifichistorical attribute for the end user based on the received informationindicative of the identity of the end user, wherein the user specifichistorical attribute is indicative of at least one of a previous headmovement speed for the end user, a previous head movement accelerationfor the end user, and a previous eye movement to head movementrelationship for the end user.

The virtual object may be at least one of a virtual text object, avirtual numeric object, a virtual alphanumeric object, a virtual tagobject, a virtual field object, a virtual chart object, a virtual mapobject, a virtual instrumentation object or a virtual visualrepresentation of a physical object.

Another embodiment is directed to a method of operation in an augmentedreality system, the method comprising receiving information indicativeof an identity of the end user, retrieving at least one user specifichistorical attribute for the end user based at least in part on thereceived information indicative of the identity of the end user, andproviding frames to the end user based at least in part on the retrievedat least one user specific historical attribute for the end user. Thereceived information may be image information indicative of an image ofat least a portion of an eye of the end user.

The retrieved at least one user specific historical attribute for theend user may be at least one attribute that provides an indication of atleast one head movement attribute for the end user, wherein the headmovement attribute is indicative of at least one previous head movementof the end user. Or the retrieved at least one user specific historicalattribute for the end user may be at least one attribute that providesan indication of at least one previous head movement speed for at leastone previous head movement for the end user. Or, the retrieved at leastone user specific historical attribute for the end user may be at leastone attribute that provides an indication of variation in a headmovement speed across at least part of a range of at least one previoushead movement by the end user. Or, the retrieved at least one userspecific historical attribute for the end user may be at least oneattribute that provides an indication of at least one previous headmovement acceleration for at least one previous head movement by the enduser. Or, the retrieved at least one user specific historical attributefor the end user may be at least one attribute that provides anindication of a relationship between at least one previous head movementand at least one previous eye movement by the end user. Or, theretrieved at least one user specific historical attribute for the enduser may be at least one attribute that provides an indication of aratio between at least one previous head movement and at least oneprevious eye movement by the end user.

The method may further comprise predicting at least an end point of ahead movement of the end user, and providing frames to the end userbased at least in part on the retrieved at least one user specifichistorical attribute for the end user includes rendering at least onesubsequent frame to at least one image buffer, the at least onesubsequent frame shifted toward the predicted end point of the headmovement.

The method may, further comprise rendering a plurality of subsequentframes that shift toward the predicted end point of the head movement inat least partial accommodation of at least one head movement attributefor the end user, the head movement attribute indicative of at least oneprevious head movement of the end user.

The head movement attribute indicative of at least one previous headmovement of the end user may be a historical head movement speed, ahistorical head movement acceleration for the end user or a historicalratio between head movement and eye movement for the end user.

The method may further comprise predicting an occurrence of a headmovement of the end user based at least in part on a location ofappearance of the virtual object in the field of view of the end user.The location of appearance of the virtual object may be determined inthe same manner described above.

Another embodiment is directed to detecting an indication that a spacingas presented to an end user between some pixels in a frame will differfrom a spacing between other pixels in the frame, adjusting a first setof pixels based on the detected indication, and providing at least aportion of at least one subsequent frame with the adjusted first set ofpixels to at least partially compensate for the difference in spacing aspresented to the end user. The pixel characteristics (e.g., size,intensity, etc.) may be perceptible to the end user.

The method may further comprise selecting a first set of pixels of theframe based on a direction of the detected head movement, wherein thedirection of the first set of pixels is the same as the direction of thedetected head movement, and increasing a size of the first set of pixelsof the at least one subsequent frame. The method may further compriseselecting a first set of pixels of the frame based on a direction of thedetected head movement wherein the direction of the first set of pixelsis the same as the direction of the detected head movement, andincreasing an intensity of the first set of pixels of the at least onesubsequent frame in response to the detected head movement.

The method may further comprise selecting a first set of pixels of theframe based on a direction of the detected head movement wherein thedirection of the first set of pixels is the opposite as the direction ofthe detected head movement, and decreasing a size of the first set ofpixels of the at least one subsequent frame in response to the detectedhead movement.

The method may further comprise selecting a first set of pixels of theframe based on a direction of the detected head movement wherein thedirection of the first set of pixels is the opposite as the direction ofthe detected head movement, and decreasing an intensity of the first setof pixels of the at least one subsequent frame in response to thedetected head movement.

Another embodiment id directed to a method of operation in a virtualimage presentation system, the method comprising rendering a firstcomplete frame to an image buffer, wherein the first complete frameincludes pixel information for sequential presentation of pixels to forman image of a virtual object, starting a presentation of the firstcomplete frame, and dynamically interrupting the presenting of the firstcomplete frame before completion of the presentation of the firstcomplete frame by a presentation of an update to the first completeframe in which a portion of the pixel information has changed from thefirst complete frame.

Another embodiment is directed to a method of operation in a virtualimage presentation system, the method comprising rendering a firstcomplete frame having a first field and a second field to an imagebuffer, wherein the first field includes at least a first spiral scanline and the second field includes at least a second spiral scan line,the second spiral scan line interlaced with at least the first spiralscan line, reading out of the frame buffer which stores the firstcomplete frame, and dynamically interrupting the reading out of thefirst complete frame before completion of the reading of the firstcomplete frame by a reading out of an update to the first complete framein which a portion of the pixel information has changed from the firstcomplete frame. The dynamic interruption of the reading out may be basedon a detected head movement of an end user, wherein the detected headmovement exceeds a nominal head movement value.

Another embodiment is directed to a method of operation in a virtualimage presentation system, the method comprising rendering a firstcomplete frame having a first field and a second field to an imagebuffer, wherein the first field includes at least a first Lissajous scanline and the second field includes at least a second Lissajous scanline, the second Lissajous scan line interlaced with at least the firstLissajous scan line, reading out of the frame buffer which stores thefirst complete frame, and dynamically interrupting, based on a detectedhead movement of an end user exceeding a nominal head movement value,the reading out of the first complete frame before completion of thereading of the first complete frame by a reading out of an update to thefirst complete frame in which a portion of the pixel information haschanged from the first complete frame. The method may further comprisephase shifting the Lissajous scan lines to interlace the Lissajous scanlines.

Another embodiment is directed to a method of operation in a virtualimage presentation system, the method comprising for each of a pluralityof frames, determining a respective resolution for each of at least twoportions of the respective frame in response to a detected head movementof an end user, and presenting the virtual objects based on thedetermined respective resolutions of the at least two portions of therespective frame. The portion of the respective frame may be at leastone of a field of the frame, a line of the frame, a pixel of the frame.The method may further comprise adjusting a characteristic of a drivesignal between presenting a first portion of a the frame and a secondportion of the frame to create a variable resolution in the image of thevirtual object. The characteristic of the drive signal may be at leastone of an amplitude of the drive signal and a slope of the drive signal.

The method may further comprise assessing a point of attention in atleast a first image for the end user, based on at least one of aprocessed eye tracking data, a determined location of appearance of avirtual object in a field of view of the end user relative to an enduser frame of reference, a determined location of appearance of thevirtual object when newly introduced in the field of view of the enduser, and a determined location of appearance of the virtual object in anew position in an image relative to a position of the virtual object inat least one previous image.

The method may further comprise increasing the resolution in at leastone subsequent image in a portion of the at least one subsequent imagethat is at least proximate to the assessed point of attention relativeto other portions of the at least one subsequent image. The method mayfurther comprise decreasing the resolution in at least one subsequentimage in a portion of the at least one subsequent image that is distalto the assessed point of attention relative to other portions of the atleast one subsequent image.

Another embodiment is directed to a method of operation in a virtualimage presentation system, the method comprising displaying at least onevirtual object to an end user, and temporarily blanking a portion of thedisplay of the at least one virtual object when at least one of adetected head movement exceeds a nominal head movement value and apredicted head movement is predicted to exceed a head movement value.The method may further comprise processing head tracking data suppliedvia at least one transducer to determine the at least one of thedetected head movement and the predicted head movement, wherein the headtracking data indicative of at least an orientation of a head of the enduser.

Another embodiment is directed to a projector apparatus to project atleast virtual images in an augmented reality system, the projectorapparatus comprising a projector element, a support that supports theprojector element with the projector element moveable in at least oneaxis of freedom, at least one actuator coupled to selectively move theprojector element, and a control subsystem communicatively coupled tocontrol the actuator such that the projector element is moved inresponse to at least one of a detection of a head movement of an enduser that exceeds a nominal head movement value and a prediction of ahead movement of the end user that is predicted to exceed the nominalhead movement value. The projector element may further comprise at leasta first optical fiber, the first optical fiber having a back end and afront end, the back end coupled to receive images, the front endpositioned to transmit images therefrom.

The support element may comprise a piezoelectric collar that receives atleast the first optical fiber proximate but spaced rear-wardly from thefront end of the first optical fiber such that a portion of the firstoptical fiber proximate the front end thereof extends from thepiezoelectric collar and is free to oscillate with a defined resonancefrequency.

The projector apparatus of claim 87, wherein the at least on controlsubsystem is communicatively coupled to receive head tracking datasupplied via at least one transducer, the head tracking data indicativeof at least an orientation of a head of the end user. The controlsubsystem, for each of at least some of a plurality of images presentedto the end user, determines a location of appearance of a virtual objectin a field of view of the end user relative to an end user frame ofreference, assesses whether the determined location requires the enduser to turn a head of the end user, and predicts the occurrence of thehead movement based on the assessment.

Another embodiment is directed to a method of operation in a virtualimage presentation system, the method comprising over-rendering a framefor a defined field of view such that a pixel information for a set ofpixels of the frame exceeds the maximum area of display at the maximumresolution, determining a portion of the frame to present to the enduser based on at least one of a detected head movement and a predictedhead movement, and selectively reading out only the determined portionof the frame.

Another embodiment is directed to a user display device, comprising ahousing frame mountable on a head of a user, a lens mountable on thehousing frame, and a projection subsystem coupled to the housing frameto determine a location of appearance of a display object in a field ofview of the user based at least in part on at least one of a detectionof a head movement of the user and a prediction of a head movement ofthe user, and to project the display object to the user based on thedetermined location of appearance of the display object. The location ofappearance of the display object may be moved in response to the atleast one of the detection of the head movement of the user orprediction of the head movement of the user that exceeds or it predictedto exceed a nominal head movement value. The prediction of the headmovement of the user may be based on a prediction of a user's shift infocus or on a set historical attributes of the user.

The user display device may further comprise a first pair of camerasmountable on the housing frame to track a movement of the user's eyesand estimate a depth of focus of the user's eyes based on the trackedeye movements. The projection subsystem may project the display objectbased on the estimated depth of focus.

The user display device may further comprise a second pair of camerasmountable on the housing frame to capture a field of view image as seenby the user's eyes, wherein the field of view image contains at leastone physical object. The projection sub system may project the displayobject in a manner such that the display object and the physical objectcaptured through the second pair of cameras are inter-mixed and appeartogether in the same frame. The location of appearance may be based atleast in part on the physical object. The display object and thephysical object may have a predetermined relationship. The capturedfield of view image may be used to gather information regardingmovements of the head of the user, wherein the information regardingmovements of the head of the user comprises a center of attention of theuser, an orientation of the head of the user, a direction of the head ofthe user, a speed of movement of the head of the user, an accelerationof the head of the user and a distance of the head of the user inrelation to a local environment of the user.

The lens may comprise at least one transparent surface to selectivelyallow a transmission light such that the user is able to view a localenvironment. The projection subsystem may project the display object ina manner such that the user views both the display object and the localenvironment as viewed through the transparent surface of the lens.

The user display device may further comprise at least one intertialtransducer to capture a set of intertial measurements indicative ofmovement of the head of the user, wherein the set of intertialmeasurements comprises a speed of movement of the head of the user, anacceleration of movement of the head of the user, a direction ofmovement of the head of the user, a position of the head of the user andan orientation of the head of the user.

The user display device may further comprise at least one light sourceto illuminate at least one of the head of the user and a localenvironment of the user.

The projection sub system may adjust at least one of a perceived size,an intensity and a resolution of a set of pixels associated with thedisplay object to compensate for the at least one of the detected headmovement and the predicted head movement. The display object may be oneof a virtual object and an augmented virtual object.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of using predictive head tracking forrendering frames to an end user.

FIG. 2 illustrates an example of a technique that predicts head movementbased on characteristics of virtual objects presented to the end user.

FIG. 3 illustrates an example where a center of the frame is shifted.

FIG. 4 illustrates an example of a technique that predicts head movementbased on a set of historical attributes of the end user.

FIG. 5 illustrates another example of the technique that predicts headmovement based on historical attributes.

FIG. 6 illustrates an example of retrieving various historicalattributes of the user

FIG. 7 illustrates an example of rendering a subsequent frame based on apredicted end point.

FIG. 8 illustrates another example of rendering the subsequent frame.

FIG. 9 illustrates an example of predicting an occurrence of headmovement.

FIG. 10 illustrates an example of adjusting pixels based on headmovement.

FIG. 11 illustrates an example of rendering frames with adjusted pixels.

FIG. 12 illustrates an example of increasing a size and/or intensity ofpixels.

FIG. 13 illustrates an example of dynamically interrupting apresentation of a frame.

FIG. 14 illustrates an example of presenting a portion of an updatedframe.

FIG. 15 illustrates an example of reading an update frame.

FIG. 16 illustrates an example of phase shifting.

FIG. 17 illustrates an example of causing variable resolution within animage.

FIG. 18 illustrates an example of adjusting an amplitude of a drivesignal.

FIG. 19 illustrates an example of adjusting a resolution in a subsequentimage based on the end user's point of attention.

FIG. 20 illustrates another example of adjusting the resolution.

FIG. 21 illustrates an example of determining a location of appearanceof a virtual object.

FIG. 22 illustrates an example of blanking a portion of displaying avirtual object.

FIG. 23 illustrates an example of predicting head movement based onattractiveness of virtual object.

FIG. 24 illustrates an example of strobing.

FIG. 25 illustrates an example of selectively activating an actuator tomove a projector element.

FIG. 26 illustrates an example of selectively reading out portions of aframe.

FIG. 27 illustrates an example of selectively reading out portions basedon a determined location of a virtual object.

FIG. 28 illustrates another example of selectively reading out portions.

FIG. 29 illustrates an example of determining a portion of an image topresent to the end user.

FIG. 30 illustrates an example of dynamically addressing a portion of anover-rendered frame.

FIG. 31 illustrates an example of a frame having pixel information.

FIG. 32 illustrates an example of a raster scan pattern.

FIG. 33 illustrates an example of a spiral scan pattern.

FIG. 34 illustrates an example of a Lissajous scan pattern.

FIG. 35 illustrates an example of a multi-field spiral scan pattern.

FIG. 36A illustrates an example of a distortion of a raster scan patternduring rapid lateral movement of the end user's head.

FIG. 36B illustrates an example of a distortion of a raster scan patternduring vertical upward movement of the end user's head.

FIG. 37A illustrates an example of a distortion of a spiral scan lineduring rapid lateral movement of the end user's head to the left.

FIG. 37B illustrates an example of a distortion of a spiral scan lineduring very rapid lateral movement of the user's head to the left.

FIG. 38 illustrates an overview of the virtual image generation system.

DETAILED DESCRIPTION

The description that follows relates to display systems and methods tobe used in virtual reality and/or augmented reality systems. However, itis to be understood that the while the invention lends itself well toapplications in virtual reality, the invention, in its broadest aspects,may not be so limited.

Referring first to FIG. 38, FIG. 38 shows a virtual image generationsystem 3800 which may operate to provide virtual images to an end user3802, according to one illustrated embodiment.

The virtual image generation system 3800 may be operated as an augmentedreality system, providing images of virtual objects intermixed withphysical objects in a field of view of the end user. There are twofundamental approaches when operating the virtual image generationsystem 3800 as an augmented reality system. A first approach employs oneor more imagers (e.g., cameras) to capture images of the ambientenvironment. The virtual image generation system 3800 may inter-mix thevirtual images into the data representing the images of the ambientenvironment. A second approach employs one or more at least partiallytransparent surfaces through which the ambient environment can be seenand on to which the virtual image generation system 3800 produces imagesof virtual objects. As will be apparent to those of skill in the art, atleast some of the aspects described herein are particularly suited toaugmented reality systems.

The virtual image generation system 3800 may be operated as a virtualreality system, providing images of virtual objects in a virtualenvironment.

The virtual image generation system 3800, and the various techniquestaught herein, may be employed in applications other than augmentedreality and virtual reality systems. For example, various techniques maybe applied to any projection or display system. For example, the varioustechniques described herein may be applied to pico projectors wheremovement may be movement of an end user's hand rather than headmovement. Thus, while often described herein in terms of an augmentedreality system, the teachings should not be limited to such systems orsuch uses.

At least for augmented reality applications, it may be desirable tospatially position various virtual objects relative to respectivephysical objects in a field of view of an end user 3802. Virtualobjects, also referred to herein as virtual tags or tag or call outs,may take any of a large variety of forms, basically any variety of data,information, concept or logical construct capable of being representedas an image. Non-limiting examples of virtual objects may include: avirtual text object, a virtual numeric object, a virtual alphanumericobject, a virtual tag object, a virtual field object, a virtual chartobject, a virtual map object, a virtual instrumentation object or avirtual visual representation of a physical object.

Head tracking accuracy and latency have been problems for virtualreality and augmented reality systems. Tracking inaccuracies and latencyproduce inconsistency between the end user's visual system andvestibular system. Such may lead to queasiness and discomfort. Such isparticularly problematic in display systems that fill a large portion ofthe end user's field of view. Approaches to addressing such may includeincreasing frame rate or effective frame rate, for instance via strobingor flashing or via other techniques. As described herein, predictivehead tracking may be employed to address such, for instance by reducinglatency. Predictive head tracking may rely on any of a large variety offactors or approaches, including historical data or attributes for aspecific end user. Also as described herein, blanking of display orpresentation may be effectively employed, for instance blanking duringrapid head movements.

At least for augmented reality applications, placement of virtualobjects in spatial relation to physical objects (e.g., presented toappear spatially proximate a physical object in two- orthree-dimensions) may be a nontrivial problem. For example, headmovement may significantly complicated placement of virtual objects in aview of an ambient environment. Such is true whether the view iscaptured as an image of the ambient environment and then projected ordisplayed to the end user 3802, or whether the end user 3802 perceivesthe view of the ambient environment directly. For instance, headmovement will like cause a field of view of the end user 3802 to change,which will likely require an update to where various virtual objects aredisplayed in the field of view of the end user 3802. Additionally, headmovements may occur within a large variety of ranges and speeds. Headmovement speed may vary not only between different head movements, butwithin or across the range of a single head movement. For instance, headmovement speed may initially increase (e.g., linearly or not) from astarting point, and may decrease as a ending point is reached, obtaininga maximum speed somewhere between the starting and ending points of thehead movement. Rapid head movements may even exceed the ability of theparticular display or projection technology to render images that appearuniform and/or as smooth motion to the end user 3802.

In the embodiment illustrated in FIG. 38, the virtual image generationsystem 3800 includes a projection subsystem 3804 operable to projectimages on a partially transparent display surface 3806 which ispositioned in the end user's 3802 field of view between the eyes 3808 ofthe end user 3802 and an ambient environment. The virtual imagegeneration system 3800 may be worn or mounted on a head 3810 of the enduser 3802, for example incorporated into a pair of glasses or a visor.

In the illustrated embodiment, the projection subsystem 3804 includesone or more optical fibers 3812 (e.g., single mode optical fiber) whichhave a back or distal end 3812 a into which light is received and afront or proximate end 3812 b from which light is provided to thepartially transparent display surface 38063806 or projected directlyinto the eyes 3808 of the end user 3802. The projection subsystem 3804may also include one or more light sources 3815 that produces the light(e.g., emits light of different colors in defined patterns), andcommunicatively couples the light to the back or distal end 3812 a ofthe one or more optical fibers 3812. The light source(s) 3815 may takeany of a large variety of forms, for instance a set of RGB lasers (e.g.,laser diodes capable of outputting red, green and blue light) operableto respectively produce red, green and blue coherent collimated lightaccording to a defined pixel patterns specified in respective frames ofpixel information or data. Laser light provides high color saturationand are highly energy efficient.

While FIG. 38 shows a single optical fiber 3812, some implementationsmay employ two or more optical fibers 3812, breaking the light up intomultiple channels. In such implementations, the optical fibers 3812 mayhave staggered tips or beveled and polished tips to bend the light,reducing optical spacing between the channels. The optical fibers 3812may be conveniently packaged as a ribbon cable. Suitable optics mayproduce a conjugate of the respective images produced by each of thechannels.

The one or more optical fibers 3812 may be supported by a yoke 3814 witha portion of the front or proximate end 3812 b extending therefrom. Theyoke 3814 may be operable to set the front or proximate end 3812 b inoscillatory motion. For example, the yoke 3814 may comprise a tube of apiezoelectric transducer 3814 a (only one shown in FIG. 38). A number ofelectrodes 3813 (e.g., four illustrated, only one called out) areradially arranged about the piezoelectric transducer 3814 a. Applyingcontrol signals, e.g., via frame buffer 3828, to respective electrodes3813 associated with the piezoelectric transducer 3814 a can cause thefront or proximate end 3812 b of the optical fiber(s) 3812 to oscillatevibrate in a first resonance mode. A size of vibrations or amount oftravel off center is controllable via the applied drive signals toobtain any of a variety of at least bi-axial patterns. Patterns may, forinstance, include a raster scan pattern, spiral or volute scan pattern,or a Lissajous or figure 8 scan pattern.

FIG. 31 shows a frame 3100 of pixel information or data that specifiespixel information or data to present an image, for example, an image ofone or more virtual objects, according to one illustrated embodiment.The frame 3100 is schematically illustrated with a cell 3100 a,-3100 n(only two called out, collectively 3102) each pixel. Sequences of cellsarranged in rows or lines 3104 a, 31004 b-3100 n (three called out,collectively 3104), illustrated as extending horizontally across thedrawing sheet in FIG. 31. The frame 3100 includes a plurality of lines3104. FIG. 31 employs ellipses to represent missing information, such ascells or lines that have been omitted for clarity of illustration.

Each cell 3102 of the frame 3100 may specify values (collectively 3106)for each of a plurality of colors for the respective pixel to which thecell corresponds and/or intensities. For instance, the frame 3100 mayspecify one or more values for red 3106 a, one or more values for green3106 b and one or more values for blue 3106 c for each pixel. The values3106 may be specified as binary representations for each of the colors,for instance a respective 4 bit number for each color. Each cell 3102 ofthe frame 3100 may additionally include an amplitude or radial value#P06d that specifies an amplitude or radial dimension for each pixel,for example where the frame 3100 may be used with a spiral scan linepattern based system or with a Lissajous scan line pattern based system.

The frame 3100 may include one or more fields, collectively 3110. Theframe 3100 may consist of a single field. Alternatively, the frame 3100may comprise two, or even more fields 3110 a-3110 b. The frame 3100illustrated in FIG. 31 shows two fields 3110 a-3110 b. The pixelinformation for a complete first field 3110 a of the frame 3100 may bespecified before the pixel information for the complete second field3110 b, for example occurring before the pixel information for thesecond field 3110 b in an array, an ordered list or other data structure(e.g., record, linked list). A third or even a fourth field may followthe second field 3110 b, assuming a presentation system is configured tohandle more than two fields 3110 a-3110 b.

FIG. 32 schematically represents a raster scan pattern 3200. In theraster scan pattern 3200, pixels 3202 (only one called out) aresequentially presented. The raster scan pattern 3200 typically presentspixels from left to right (indicated by arrows 3204 a, 3204 b, then fromtop to bottom (indicated by arrow 3206). Thus, the presentation maystart at the upper right corner and traverse left across a first line3208 a until the end of the line is reached. The raster scan pattern3200 typically then starts from the left in a next line down. Thepresentation may be temporarily blacked out or blanked which returningfrom the end of one line to the start of the next line. This processrepeats line-by-line until the bottom line 3208 n is completed, forexample at the bottom right most pixel. With the frame 3100 beingcomplete, a new frame is started, again returning the right of the topmost line of the next frame. Again, the presentation may be blankedwhile returning from the bottom left to the top right to present thenext frame.

Many implementations of raster scanning employ what is term as aninterlaced scan pattern. In interlaced raster scan patterns, lines fromthe first and the second fields 3210 a, 3210 b are interlaced. Forexample, when presenting lines of the first field 3210 a, the pixelinformation for the first field 3210 a may be used for the odd numberedlines only, while the pixel information for the second field 3210 b maybe used for the even numbered lines only. Thus, all of the lines of thefirst field 3210 a of the frame 3100 (FIG. 31) are typically presentedbefore the lines of the second field 3210 b. The first field 3210 a maybe presented using the pixel information of the first field 3210 a tosequentially present line 1, line 3, line 5, etc. Then the second field3210 b of the frame 3100 (FIG. 31) may be presented following the firstfield 3210 a, by using the pixel information of the second field 3210 bto sequentially present line 2, line 4, line 6, etc.

FIG. 33 schematically represents a spiral scan pattern 3300, accordingto one illustrated embodiment. The spiral scan pattern 3300 may consistof a single spiral scan line 3302, which may include one or morecomplete angular cycles (e.g., 360 degrees) which may be denominated ascoils or loops. The pixel information is used to specify the colorand/or intensity of each sequential pixel, as the angle increments. Anamplitude or radial value 3208 (FIG. 31) specifies a radial dimension#R06 from a starting point 3308 of the spiral scan line 3302.

FIG. 34 schematically represents a Lissajous scan pattern 3400,according to one illustrated embodiment. The Lissajous scan pattern 3400may consist of a single Lissajous scan line 3402, which may include oneor more complete angular cycles (e.g., 360 degrees) which may bedenominated as coils or loops. Alternatively, the Lissajous scan pattern3400 may include two or more Lissajous scan lines 3402, each phaseshifted with respect to one another to nest the Lissajous scan lines3402. The pixel information is used to specify the color and/orintensity of each sequential pixel, as the angle increments. Anamplitude or radial value 3208 (FIG. 31) specifies a radial dimensionfrom a starting point of the Lissajous scan line 3402.

FIG. 35 schematically represents a multi-field spiral scan pattern 3500,according to one illustrated embodiment. The multi-field spiral scanpattern 3500 includes two or more distinct spiral scan lines,collectively 3502, FIG. 35 illustrating four spiral scan lines 3502a-3502 d. The pixel information for each spiral scan 3502 line may bespecified by a respective field (e.g., 3210 a, 3210 b) of a frame 3100(FIG. 31). Advantageously, multiple spiral scan lines 3502 may be nestedsimply by shifting a phase between each successive ones of the spiralscan lines 3502. The phase difference between spiral scan lines 3502should be a function of the total number of spiral scan lines 3502 whichwill be employed. For example, four spiral scan lines 3502 a-3502 d maybe separate by a 90 degree phase shift. An exemplary embodiment mayoperate at a 100 Hz refresh rate with 10 distinct spiral scan lines(i.e., subspirals). Similar to the embodiment of FIG. 33, one or moreamplitude or radial values 3208 (FIG. 31) specify a radial dimension3506 from a starting point 3508 of the spiral scan lines 3502.

As is evident from FIGS. 34 and 35, relative spacing between adjacentpixels may vary throughout an image. It may be advantageous to at leastpartially accommodate or compensate for this non-uniformity. Forexample, it may be advantageous to adjust pixel size, for instanceincreasing perceived pixel size for pixels that are spaced farther apartthan other pixels. Such may, for instance, be implemented via selectiveblurring (e.g., variable focus lens, variable diffuser, jitter) toincrease Gaussian spot size. Additionally or alternatively, it may beadvantageous to adjust intensity for pixels that are spaced fartherapart than other pixels.

Returning to FIG. 38, driving the piezoelectric transducer 3814 a withsine wave drive signals at a resonant frequency about a first axis andat a resonance frequency about a second axis, perpendicular to the firstaxis, produces a spiral scan pattern. The spiral scan pattern may becharacterized by a radial dimension that varies as an angular dimensionvaries. For example, a radial dimension may vary linearly, ornonlinearly, while the radial dimension varies from 0 degrees to, orthrough, 360 degrees. In appearance, the spiral scan line may appear asa continuous spiral, starting at a start point and sweeping radiallyoutward while rotating in a plane. Each complete angular cycle may bedescribed as constituting a coil or loop. Spiral scan lines may bedefined has having any desired number of coils or loops before startingover at the start point. A refresh period in which display orpresentation is blanked may occur between an end of a temporally firstspiral scan pattern and a stat of a next temporally successive spiralscan pattern. An outer most radial dimension of the spiral scan patternmay be set by amplitude modulating of the sine wave drive signal.Amplitude modulation of a spiral scan line pattern adjusts the radialdimension without affecting the angular dimension.

Thus, amplitude modulation will not affect the frequency of cycles(e.g., number of coils or loops) or number of cycles in a given time fora given scan line. The position of the front or proximate end 3812 b inthe pattern is synchronized with the output of the light source(s) 3815to form two- or three-dimensional images.

While not illustrated, the projection subsystem 3804 may include one ormore optical components (e.g., lenses, filters, gratings, prisms,reflectors, dichroic reflectors, defractors) that direct the output fromthe front or proximate end 3812 b of the one or more optical fibers 3812directly or indirectly toward the eyes 3808 of the end user 3802, forexample via partially transparent display surface 3806. While notillustrated, the projection subsystem 3804 may include one or moreoptical components that modulate a depth of Z-axis position of pixeldata. Such may, for example, take the form of a flexible reflective(e.g., nitride sputter coated with aluminum) membrane and one or moreelectrodes operated to cause deflection of the flexible reflectivemembrane. The flexible reflective membrane is positioned to reflect andfocus light emitted from the front or proximate end 3812 b of the one ormore optical fibers 3812. The flexible reflective membrane isselectively operable based on depth map for the pixel data orinformation to focus light in the Z-dimension or axis. The flexiblereflective membrane may employ Gaussian spots to produce an appearanceof depth, certain virtual objects in an image appearing in focus whileothers appearing out of focus. Additionally or alternatively, the systemmay employ one or more Kerr effect lens.

While not necessary to a head worn embodiment, the optical fibers 3812,and optionally the yoke 3814, may be supported for movement in one ormore directions. For example, the optical fibers 3812, and optionallythe yoke 3814, may be supported via gimbals 3816 for 2, 3 or moredegrees of freedom of movement. The gimbals 3816 may include a turntable3816 a, a first actuator 3818 a (e.g., electric motor, solenoid,piezoelectric transducer) operable to pivot or rotate about a first axis3820 a. The gimbals 3816 may include a bracket 3816 b supported by aframe 3816 c on the turntable 3816 a, a second actuator 3818 b (e.g.,electric motor, solenoid, piezoelectric transducer) operable to pivot orrotate about a second axis 3820 b. The gimbals 3816 may include a shaft3816 d pivotally supported by the bracket 3816 b, a third actuator 3818c (e.g., electric motor, solenoid, piezoelectric transducer) operable topivot or rotate about a third axis 3820 c. The first, second and thirdaxes (collectively 3820) may be orthogonal axes.

In the embodiment illustrated in FIG. 38, the virtual image generationsystem 3800 includes a control subsystem 3822. The control subsystem3822 may take any of a large variety of forms, one of which isillustrated in FIG. 38.

The control subsystem 3822 includes a number of controllers, forinstance one or more microcontrollers, microprocessors or centralprocessing units (CPUs) 3824, digital signal processors (DSPs), graphicsprocessing units (GPUs) 3826, other integrated circuit controllers suchas application specific integrated circuits (ASICs), programmable gatearrays (PGAs) for instance field PGAs (FPGAs), and/or programmable logiccontrollers (PLUs). In the embodiment illustrated in FIG. 38, themicroprocessor 3824 controls overall operation, while the GPU 3826renders frames (e.g., sets of pixel data) to one or more frame buffers3828 a-3828 n (collectively 3828). While not illustrated, one or moreadditional integrated circuits may control the reading into and/orreading out of frames from the frame buffer(s) 3828 and operation of thepiezoelectric transducers or electrodes 3814 a, synchronizing both toproduce two- or three dimensional images. Reading into and/or out of theframe buffer(s) 3828 may employ dynamic addressing, for instance whereframes are over rendered.

The control subsystem 3822 includes one or more nontransitory computer-or processor-readable media to store instructions and data. Thenontransitory computer- or processor-readable media may for exampleinclude the frame buffer(s) 3828. The nontransitory computer- orprocessor-readable media may, for example, include one or morenonvolatile memories, for instance read only memory (RAM) 3830 or flashmemory. The nontransitory computer- or processor-readable media may, forexample, include one or more volatile memories, for instance randomaccess memory (RAM) 3832. The control subsystem 3822 may include othervolatile and nonvolatile memory, include spinning media storage as wellas solid state storage devices.

In implementations where the actuators (collectively 3818) are employed,the control subsystem 3822 may optionally include one or more dedicatedmotor controllers 3834 communicatively coupled to drive the actuators3818 via motor control signals.

The control subsystem 3822 may optionally include one or morecommunications ports 3836 a, 3836 b (collectively 3836) that providecommunications with various other systems, components or devices. Forexample, the control subsystem 3822 may include one or more wiredinterfaces or ports 3836 a which provide wired or opticalcommunications. Also for example, the control subsystem 3822 may includeone or more wireless interfaces or ports such as one or more radios(i.e., wireless transmitter, receiver, transceiver) 3836 b which providewireless communications.

As illustrated, the wired interfaces or ports 3836 a provide wired oroptical communications with an environmental imaging system 3838 includeone or more cameras 3838 a positioned and oriented to capture images ofan environment in which the end user 3802 is located. Such may be usedto sense, measure or collect information about the end user 3802 and/orthe environment. For instance, such may be used to detect or measuremovements and/or positions of the end user 3802 or parts of the enduser's 3802 body, such as the head 3810. As illustrated, the wiredinterfaces or ports 3836 a may optionally provide wired or opticalcommunications with a structure lighting system 3840 which includes oneor more light sources 3840 a positioned and oriented to illuminate theend user 3802, a portion of the end user 3802 such as the head 3810and/or the environment in which the end user 3802 is located.

As illustrated, the wireless interfaces or ports 3836 b provide wireless(e.g., RF, microwave, IR) communications with one or more head worntransducer system 3842 that includes one or more inertial transducers3842 a to capture inertial measures indicative of movement of the head3810 of the end user 3802. Such may be used to sense, measure or collectinformation about head movements of the end user 3802. For instance,such may be used to detect or measure movements, speeds, acceleration,and/or positions of the head 3810 of the end user 3802. As illustrated,the wired interfaces or ports 3836 a may optionally provide wired oroptical communications with an imaging system 3842 including, forexample, one or more forward facing imagers or cameras 3842 a. Such maybe used to capture information about the environment in which the enduser 3802 is located. Such may be used to capture information indicativeof distance and orientation of the end user 3802 with respect to thatenvironment and specific objects in that environment. When head worn,the forward facing imagers or cameras 3842 a are particularly suited tocapture information indicative of distance and orientation of the enduser's head 3810 with respect to the environment in which the end user3802 is located and specific objects in that environment. Such may, forexample be employed to detect head movement, speed and/or accelerationof head movements. Such may, for example, be employed to detect or infera center of attention of the end user 3802, for example based at leastin part on an orientation of the end user's head 3810. Orientation maybe detected in any direction (e.g., up/down, left/right with respect toreference frame of end user).

In some implementations all communications may be wired, while in otherimplementations all communications may be wireless. In still furtherimplementations the choice of wired and wireless communications may bedifferent from that illustrated in FIG. 38. Thus, the particular choiceof wired or wireless communications should not be considered limiting.

Various components of the control subsystem 3822, for example themicroprocessor 3824, GPU 3826, frame buffer(s) 3828, ROM 3830, RAM 3832,and/or optionally dedicated motor controller(s) 3834 may becommunicatively coupled via one or more communications channels, forinstances one or more buses 3846 (only one illustrated). The buses 3846may take a variety of forms including instruction buses, data buses,address buses, other communications bus, and/or power buses.

The ability to predict head movements allows a virtual image generationsystem 3800 (FIG. 38), such as an augmented reality system, to quicklyupdate the presentation of images and/or to accommodate or compensatefor head movement. For example, subsequent frames may be rendered orread out earlier than would be possible if only sensed head movementswere employed. As will be apparent from the discussions herein,accommodation or compensation may take a variety of forms. For example,subsequent frames may be rendered or read out with a shifted field ofview or a center that is shifted toward or to an area of attention orfocus of the end user. Also for example, subsequent frames may berendered or read out to accommodate or compensate for variationresulting from the head movement. For instance, in certain display orprojection technologies (e.g., “flying pixel” technologies where pixelsare displayed sequentially, such as raster scan, spiral scan, Lissajousscan), rapid head movement may cause a change in spacing between pixelsas a frame is presented to the end user. The accommodation orcompensation may include accommodating or compensating for thisvariation in pixel spacing. For instance, a size or perceived size ofsome pixels may be adjusted relative to other pixels. Also for instance,an intensity or perceived brightness of some pixels may be adjustedrelative to other pixels. As a further example, subsequent frames may berendered or read out with a variable resolution between differentportions of a resulting image. Other accommodation or compensationtechniques will be apparent from the this discussion. In other aspects,many of these same techniques may be employed for purposes other thanaccommodation or compensation, and may be employed independently ofpredictive head tracking, sensed head tracking, and/or with display orprojection technologies that are not “flying pixel” based.

End user movement, for example head movements may have a substantialeffect on images. As the augmented reality system attempts to renderframes subsequently frames consistent with the head movement, theresulting images of virtual objects may become compressed, expanded orotherwise distorted. This is at least partially the result of the factthat for many display or presentation technologies (i.e., “flying pixel”technologies), complete images for any given frame are not presented ordisplayed simultaneously, but rather are presented or displayed pixel bypixel. Thus, there is not a true instantaneous field of view for thesedisplay or presentation technologies. Such may occur, in differentforms, across many different types of image generation technologies, forinstance raster scan, spiral scan or Lissajous scan approaches. One ormore “white” or blank frames or images may alleviate some of the effectsof rapid head movement.

For example, FIG. 36A shows an exemplary distortion in a raster scan3600 a produced during rapid lateral movement of an end user's head. Thedistortion is likely to be nonlinear since head motion may speed upafter initiation and slow down prior to termination. The distortion is afunction of the direction, speed and acceleration of head movement andthe direction of raster scan pixel generation (e.g., right to left, topto bottom).

Also for example, FIG. 36B shows an exemplary distortion in a rasterscan 3600 produced during vertically upward movement of an end user'shead. The distortion is likely to be nonlinear since head motion mayspeed up after initiation and slow down prior to termination. Thedistortion is a function of the direction, speed and acceleration ofhead movement and the direction of raster scan pixel generation (e.g.,right to left, top to bottom).

As yet another example, FIG. 37A shows an exemplary distortion in aspiral scan line 3700 a produced during rapid lateral movement of an enduser's head to the left. The distortion is likely to be nonlinear sincehead motion may speed up after initiation and slow down prior totermination. The distortion is a function of the direction, speed andacceleration of head movement and the direction of spiral scan pixelgeneration (e.g., clockwise, increasing radius). As illustrated spacingbetween successive loops or coils of the spiral scan line 3700 aincreases in the direction of the head movement (e.g., to the left inthe drawing sheet), and decreases in the diametrically opposed direction(e.g., to the right in the drawing sheet).

As yet a further example, FIG. 37B shows an exemplary distortion in aspiral scan line 3700 b produced during very rapid lateral movement ofan end user's head to the left. The distortion is likely to be nonlinearsince head motion may speed up after initiation and slow down prior totermination. In fact, the distortion may be highly elliptical andde-centered as illustrated in FIG. 37B. The distortion is a function ofthe direction, speed and acceleration of head movement and the directionof spiral scan pixel generation (e.g., clockwise, increasing radius). Asillustrated spacing between successive loops or coils of the spiral scanline 3700 b increases in the direction of the head movement (e.g., tothe left in the drawing sheet). Where the head movement is too rapid forthe system, the left most portion of each loop or coil may be located inthe same direction as the head movement relative to a starting point ofthe spiral scan line 3700 b, as illustrated in FIG. 37B.

One advantage of employing spiral scan patterns is that thetransformation to address into the image buffer is independent of thedirection of movement (e.g., movement of head, movement of hand forhandheld pico projector).

The system above is used in all the embodiments described below. In oneembodiment, the system may be used for predictive head tracking based onpredicting a user's shift in focus. FIG. 1 shows a method 100 ofoperation in an augmented reality system employing predictive headtracking, according to one illustrated embodiment.

At 102, the augmented reality system (e.g., a controller subsystemand/or processor thereof) presents a plurality of frames as images to anend user of the augmented reality system. The frames will typicallyinclude pixel information specifying information for producing one ormore virtual objects in a field of view. As previously noted, thevirtual objects may take any of a wide variety of virtual object formsor formats, which may visual represent physical objects or mayrepresented information, data or logical constructions. Non-limitingexamples of virtual objects may include: a virtual text object, avirtual numeric object, a virtual alphanumeric object, a virtual tagobject, a virtual field object, a virtual chart object, a virtual mapobject, a virtual instrumentation object or a virtual visualrepresentation of a physical object.

At 104, the augmented reality system selects one or more virtual objectsbased at least on input indicative of attention of the end user.

The input may be an actual selection by the end user. The selection maybe made in real time or may have been previously designated by the enduser. Thus, an end user may select a certain set of virtual instrumentsas being a type of virtual object the end user typically focuses or paysattention to over other objects.

The input may be inferred from a variety of sources. The input may berelated to the virtual objects themselves. The input may be additionallyor alternatively be related to physical objects in a field of view ofthe end user or in the field of view of a display or projector. Theinput may be additionally or alternatively be related to the end userthemselves, for example a position and/or orientation of the end userand/or a portion of end user (e.g., head, eyes), or historicalattributes. The historical attributes may be end user specific, or moregeneralized or generic. The historical attributes may be indicative of aset of defined end user characteristics. End user characteristics may,for example, include head movement speed, head movement acceleration,and/or relationship between head movement and eye movement (e.g., ratioof one to the other. The end user characteristics tracked by historicalattributes may even include indications of a tendency of a given enduser to pay attention to certain virtual objects. Such may be specifiedby virtual object type (e.g., text, charts), by recentness of virtualobject (e.g., newly appearing objects), movement of a virtual object(e.g., large shifts from image to image, fast or rapidly movement,direction of movement), and/or characteristics of the virtual object(e.g., color, brightness, size).

At 106, for each of at least some of a plurality of frames beingpresented to an end user, the augmented reality system (e.g., acontroller subsystem and/or processor thereof) determines a location ofappearance of a virtual object in a field of view of the end userrelative to an end user frame of reference. For example, the augmentedreality system may determine a location of a newly introduced virtualobject, a virtual object of a defined type, a virtual object movingrapidly or over a large distance, or a virtual object that hashistorically been a point of attention for the end user.

At 108, the augmented reality system adjusts a presentation of at leastone subsequent frame based at least in part on the determined locationof appearance of the virtual object in the field of view of the enduser. Numerous ways of adjusting appearance of the virtual object in thefield of view are discussed herein, including non-exhaustivelyaccommodation or compensation, adjusting pixel size, adjusting pixelintensity, adjusting resolution, windowing, and/or blanking or strobing.

FIG. 2 shows another method 200 of operation in an augmented realitysystem, according to one illustrated embodiment. The method 200 may beemployed in executing acts 104 and/or 106 of the method 100 of FIG. 1.

The method 200 employs techniques that predict head movement based oncharacteristics of virtual objects that are or will be presented to theend user. For example, anticipating that a newly introduced virtualobject or a virtual object who's movement (e.g., due to suddenness,speed, and/or distance) will likely attract the end user's attention,resulting in a head movement to bring the particular virtual object intoor proximate a center of the end user's field of view. Additionally, oralternatively the augmented reality system may rely on othercharacteristics of the virtual objects in assessing which are mostlikely to attract attention. For example, highly attractive (e.g.,flashing, shimmering), large, fast moving, or bright virtual objects maybe more likely to attract attention than other virtual objects.

Focusing on the case of a newly introduced virtual object, the augmentedreality system (e.g., a controller subsystem and/or processor thereof)selects and/or determines the location of appearance of a virtual objectwhen newly introduced in the field of view of the end user at #AB02. Avirtual object is considered newly introduced when not appearing inprevious (temporally) related frames presented to the end user. Inparticular, the augmented reality system relies on the fact that newlyintroduced virtual objects are like to attract the attention of the enduser relative to virtual objects that appear in immediately precedingframes. Additionally, or alternatively the augmented reality system mayrely on other characteristics of the virtual objects in assessing whichare most likely to attract attention, for example to select orprioritize among multiple newly introduced virtual objects. For example,highly attractive (e.g., flashing, shimmering), large, fast moving, orbright virtual objects may be more likely to attract attention thanother virtual objects.

Focusing on the case of a moving virtual object, the augmented realitysystem (e.g., a controller subsystem and/or processor thereof) selectsand/or determines a location of an appearance of a virtual object in anew position in a frame relative to a position of the same virtualobject in at least one previous frame at 204. Thus, a sudden shifting, aquick shifting, and/or a spatially large shifting of a position of avirtual object from one frame to one or more subsequent frames may belikely to attract the attention or focus of the end user. Additionally,or alternatively the augmented reality system may rely on othercharacteristics of the virtual objects in assessing which are mostlikely to attract attention, for example to select or prioritize amongmultiple newly introduced virtual objects. For example, highlyattractive (e.g., flashing, shimmering), large, or bright virtualobjects may be more likely to attract attention than other virtualobjects.

FIG. 3 shows a method 300 of operation in an augmented reality system,according to one illustrated embodiment. The method 300 may be employedin executing act 108 of the method 100 of FIG. 1.

At 302, the augmented reality system (e.g., a controller subsystemand/or processor thereof) presents the at least one subsequent framewith a center of the at least one subsequent frame shifted at leasttowards, if not centered on, the determined location of appearance ofthe virtual object in the field of view of the end user. The center ofthe subsequent frame(s) or image(s) may be shifted to be co-located withthe location of the selected virtual object that is predicted to attractthe end user's attention. Alternatively, the center of the subsequentframe(s) may be shifted to be proximate the location of the selectedvirtual object that is predicted to attract the end user's attention.Such may be performed in two-dimensions or in three-dimensions. Forexample, the two dimensional or three dimensional position of thevirtual object may be used to adjust a field of view of subsequentimages(s) in two- or three-dimensions, respectively. The shiftedsubsequent frame(s) or image(s) are preferably timed with the predictedhead movement of the end user. Thus, the shifted subsequent frame(s) orimage(s) should be presented to the end user as close in timing with theactual head movement as possible. As discussed herein, such may accountfor speed, acceleration, and variations in speed and acceleration.

FIG. 4 shows a method 400 of operation in an augmented reality system,according to one illustrated embodiment. The method 400 may be employedin performing the method 100 of FIG. 1.

Optionally at 402, the augmented reality system receives informationindicative of an identity of the end user. The information may take anyof a large variety of forms. For example, the information may be a username or other user identifier entered by the end user (e.g., keyed) orread from a transponder, magnetic stripe, or machine-readable symbolassociated with the end user. For example, the information may includebiometric information indicative of one or more physical characteristicsof the end user. In one particularly advantageous implementation, theaugmented reality system may receive image data that represents aportion (e.g., retina) of one or both eyes of the end user. Forinstance, the augmented reality system may project light, for examplevia one or more optical fibers, into one or both eyes of an end user.The light may be modulated, for example to increase a signal to noiseratio and/or limit heating of the eye. An image sensor may capture animage of the portion of the eye, for example via the one or more opticalfibers that project the light, the optical fiber(s) providing abi-directional path. Alternatively, a dedicated optical fiber may beemployed. As a further alternative an image sensor may be positionedproximate the eye, eliminating the use of the optical fiber(s) as areturn path to the image sensor. Certain portions of the human eye(e.g., vasculature of retina) may be considered sufficiently distinctiveto serve as a unique end user identifier.

Optionally at 404, the augmented reality system retrieves at least oneuser specific historical attribute for the end user based in thereceived information indicative of the identity of the end user. Theuser specific historical attribute(s) may be indicative of at least oneof: a previous head movement speed for the end user, previous headmovement acceleration for the end user, previous eye movement to headmovement relationship for the end user, tendencies of end user to payattention to virtual objects of certain types or with certaincharacteristics.

At 406, the augmented reality system (e.g., a controller subsystemand/or processor thereof) predicts an occurrence of a head movement ofthe end user based at least in part on the determined location ofappearance of the virtual object in the field of view of the end user.Again, the augmented reality system may rely on an attractiveness of avirtual object in predicting head movements, for example on an end userby end user basis.

The augmented reality system may employ estimated speed and/or estimatedchanges in speed or estimated acceleration to at least partiallysynchronize image presentation with the predicted head movement of theend user. The estimated change in the speed in the predicted headmovement may be based on a range extending between a first defined timeafter the start of the predicted head movement and a second defined timebefore the end of the predicted head movement.

At 408, the augmented reality system estimates at least one valueindicative of an estimated speed of the predicted head movement of theend user. The augmented reality system may estimate speed based on oneor more values, parameters or characteristics. For example, theaugmented reality system may rely on a range of movement required tomove to the end user's head to a new position to observe the selected oridentified virtual object. The augmented reality system may rely onaverage speed for a sampling of humans, or may rely on historical headmovement speeds for the particular end user. The augmented realitysystem may rely on historical attributes for the particular end user.Speeds may be represented in angular velocities.

At 410, the augmented reality system estimates at least one change inthe speed in the predicted head movement of the end user which occursover a range of head movement, between a start of the predicted headmovement and an end of the predicted head movement. The changes in speedmay occur at different increments throughout some portion of thepredicted range of movement.

At 412, the augmented reality system estimates at least one valueindicative of an estimated acceleration of the predicted head movementof the end user. The estimated acceleration may be over an entire rangeof the head movement or over only a portion thereof. The estimatedacceleration may be over discrete intervals of the range of headmovement. Estimates of acceleration may be determined for one or moreintervals at some defined duration after the start of head movement.Estimates for acceleration may be determined for one or more intervalsat some defined duration before the end of head movement. Estimatingspaced from the start and/or end points may avoid large variation inacceleration measurements.

Optionally at 414, the augmented reality system determines at least onevalue that at least partially accommodates or compensates for theestimated speed of the predicted head movement of the end user. Forexample, the augmented reality system may determine values with respecta total number of frames to present in a given time and/or values thatspecify where and/or how quickly one or more virtual objects should moveacross a scene in a series of images to be rendered and/or presented.Such may be used to render subsequent frames.

Optionally at 416, the augmented reality system renders the at least onesubsequent frame based at least in part on the at least one value thatat least partially compensates for the estimated speed of the predictedhead movement of the end user. For example, the augmented reality systemmay determine values with respect a total number of frames to present ina given time and/or values that specify where and/or how quickly one ormore virtual objects should move across a scene in a series of images tobe rendered and/or presented. Such may be used to render subsequentframes.

In another embodiment, the system may be used for predictive headtracking based on a user's historical attributes. FIG. 5 shows a method500 of operation in an augmented reality system employing predictivehead tracking, according to one illustrated embodiment.

The augmented reality system may employ historical attributes inperforming predictive head tracking. The historical attributes may beend user specific, or more generalized or generic. The historicalattributes may be indicative of a set of defined end usercharacteristics. End user characteristics may, for example, include headmovement speed, head movement acceleration, and/or relationship betweenhead movement and eye movement (e.g., ratio of one to the other. The enduser characteristics tracked by historical attributes may even includeindications of a tendency of a given end user to pay attention tocertain virtual objects.

At 502, the augmented reality system receives information indicative ofan identity of the end user. The information may take any of a largevariety of forms, for example information actively provided be the enduser, read from nontransitory storage media, read from the user (e.g.,biometric data or characteristics), or inferred from end user actions.

At 504, the augmented reality system retrieves at least one userspecific historical attribute for the end user based at least in part onthe received information indicative of the identity of the end user. Theidentity information may be received, produced or determined in any of alarge variety of ways.

At 506, the augmented reality system provides frames to the end userbased at least in part on the retrieved at least one user specifichistorical attribute for the end user. For example, the augmentedreality system may provide frames from a frame buffer to a projector ordisplay device (e.g., light source paired with one or more opticalfibers), or may render frames to a frame buffer. The augmented realitysystem may provide light via at least one optical fiber which ismoveable at least bi-axially. The augmented reality system may receiveimage information indicative of an image of at least a portion of an eyeof the end user via the at least an optical fiber which also providesthe frames to the end user.

FIG. 6 shows a method 600 of operation in an augmented reality systememploying predictive head tracking, according to one illustratedembodiment. The method 600 may be employed in executing act 504 of themethod 500 of FIG. 5.

At 602, the augmented reality system retrieves at least one historicalattribute that provides an indication of at least one head movementattribute for the end user. The head movement attribute(s) is or areindicative of at least one previous head movement of the end user. Thehistorical attributes may be stored in nontransitory media, for examplein a database or other logical construct.

At 604, the augmented reality system retrieves at least one historicalattribute that provides an indication of head movement speed for atleast one previous head movement for the end user.

At 606, the augmented reality system retrieves at least one historicalattribute that provides an indication of variation in a head movementspeed across at least part of a range of at least one previous headmovement by the end user.

At 608, the augmented reality system retrieves at least one historicalattribute that provides an indication of head movement acceleration forat least one previous head movement by the end user.

At 610, the augmented reality system retrieves at least one historicalattribute that provides an indication of a relationship between headmovement and eye movement for at least one previous head and eyemovement combination by the end user. The relationship may, for example,be represented as a ratio of a head movement value representative of atleast one previous head movement and a value representative of at leastone previous eye movement by the end user. The values may berepresentative of an amount of movement the head and eyes, respectively,for example represented as an angular change. The ratio may be a ratioof historical averages of head movements and historical averages of eyemovements by the end user. Additionally or alternatively, otherrelationships between head and eye movement may be employed, for examplespeed or acceleration.

FIG. 7 shows a method 700 of operation in an augmented reality systememploying predictive head tracking, according to one illustratedembodiment. The method 700 may be employed in executing act 506 of themethod 500 of FIG. 5.

At 702, the augmented reality system predicts at least an end point of ahead movement of the end user. For example, where the appearance of avirtual object is used to predict the head movement, the relativelocation of the particular virtual object may be used as the end point.

At 704, the augmented reality system renders at least one subsequentframe to at least one image buffer. The at least one subsequent frame isshifted at least towards, or even all the way to, the predicted endpoint of the head movement.

FIG. 8 shows a method 800 of operation in an augmented reality systememploying predictive head tracking, according to one illustratedembodiment. The method 800 may be employed in executing act 704 of themethod 700 of FIG. 7.

At 802, the augmented reality system renders a plurality of subsequentframes that are shift at least toward the predicted end point of thehead movement in at least partial accommodation of at least one headmovement attribute for the end user. The head movement attributes may beindicative of various physical traits of head movement, particularlyhistorical physical traits of head movement of the end user. The headmovement attribute(s) may, for example, include one or more of: ahistorical head movement speed for the end user, a historical headmovement acceleration for the end user, and/or a historical relationship(e.g., ratio) between head movement and eye movement for the end user.Shifting may be implemented by rendering subsequent frames with thecorresponding image shifted or a center of the corresponding imageshifted relative to images corresponding to a previous frame.

FIG. 9 shows a method 900 of operation in an augmented reality systememploying predictive head tracking, according to one illustratedembodiment. The method 900 may be employed in executing act 703 of themethod 700 of FIG. 7.

At 902, the augmented reality system predicts an occurrence of a headmovement of the end user based at least in part on an appearance of avirtual object in the field of view of the end user.

The appearance may be an appearance of a new virtual object when newlyintroduced in the field of view as presented to the end user,temporally, relative to previous frames presented as images to the enduser. Alternatively or additionally, the appearance may be an appearanceof a virtual object in a new position in the field of view as presentedto the end user, relative to a position of the virtual object aspreviously presented to the end user. The prediction may, for exampletake into account of factors. For example, the prediction may be basedin part on a size or prominence of the virtual object, an amount orpercentage of change in position, speed, suddenness acceleration orother change in the position of the virtual object.

The system may also be used for dynamic control of pixel characteristicsFIG. 10 shows a method 1000 of operation in an augmented reality system,according to one illustrated embodiment.

At 1002, the augmented reality system (e.g., a controller subsystemand/or processor thereof) detects an indication that spacing aspresented to an end user between some pixels in a frame will differ fromthe spacing between other pixels in the same frame. For example, theaugmented reality system may detect an indication that the spacing aspresented to the end user between pixels of a first set of pixels in theframe will differ from the spacing as presented to the end user betweenpixels of at least a second set of pixels in the frame. For instance,where pixels of a frame are sequentially presented (e.g., read out ofthe frame buffer) over a period of time (e.g., “flying pixel” patternssuch as raster scan pattern, spiral scan pattern, Lissajous scanpattern), rapid head movement may cause a variation in pixel spacingbetween different portions of an image or frame.

In response to a detection that the spacing between some pixels in theframe will differ as presented to the end user from the spacing of otherpixels in the frame, the augmented reality system provides at least aportion of at least one subsequent frame with at least a first set ofpixels adjusted to at least partially compensate at least one pixelcharacteristic, perceptible by an end user, of the pixels of the firstset at 1004. Such may at least partially compensate for a difference inspacing between pixels in different portions of an image as presented tothe end user.

FIG. 11 shows a method 1100 of operation in an augmented reality system,according to one illustrated embodiment. The method 1100 may be employedin executing the method 1000 of FIG. 10.

Optionally at 1102, the augmented reality system (e.g., a controllersubsystem and/or processor thereof) receives signals indicative of anoutput of at least one head worn inertial sensor worn by the end user.The inertial sensors may take a variety of forms, for example gyroscopicsensors or acceleration sensors. Inertial sensors may be single axis ormulti-axis devices. The inertial sensors may take the form of MEMSdevices.

Optionally at 1104, the augmented reality system receives signalsindicative of an output of at least one head worn imager worn by the enduser. The imager may, for example, take the form of a digital camera orother image capture device. Such may be a forward facing camera, tocapture a field of view that at least approximates a field of view ofthe end user.

Optionally at 1106, the augmented reality system detects a head movementexceeding a nominal head movement value. For example, the augmentedreality system may detect a head movement exceeding a nominal speedand/or a nominal acceleration. The augmented reality system may employsignals from the inertial sensors to detect movement, and in particularacceleration. The augmented reality system may employ signals from thehead mounted camera to detect changes in position of physical objects inthe ambient environment, particularly fixed physical objects such aswall, floors, ceilings. The augmented reality system may employ anynumber of image processing techniques. The detected change in positionallows the augmented reality system to determine a change in position ofthe head, speed of movement and acceleration. The augmented realitysystem may employ other information in addition or in place of theinertial sensor and head worn imager information. For example, theaugmented reality system may employ signals from a system that monitorsthe ambient environment and is not worn by the user, but rather tracksthe user. Such a system may employ one or more imager, for instancedigital cameras, to monitor the ambient environment. The imager(s)detect movement of the end user and parts of the end user such as thehead. Again, various image processing techniques may be employed. Such asystem may be advantageously paired with a structured light system.Alternatively, the method #CB00 may be executed independently ofdetected or even predicted head movement.

At 1108, the augmented reality system selects a first set of pixels ofthe frame, for example based on a direction of the detected headmovement. The augmented reality system may additionally select the firstset of pixels of the frame based on other criteria, for example a speedof the detected head movement.

At 1110, the augmented reality system adjusts at least one of a sizeand/or an intensity of at least some of the pixels of the first set ofpixels of at least one subsequent frame. The adjustment may be designedto at least accommodate or at least partially compensate for undesiredvariation in the frame or image that result from the head movement.

Optionally at 1112, the augmented reality system renders the at leastone subsequent frame. The rendered subsequent frame includes adjustedpixel information to at least partially accommodate or at leastpartially accommodate or compensate for undesired variation in the frameor image that result from the head movement.

Optionally at 1114, the augmented reality system reads out the at leastone subsequent frame from at least one frame buffer that stores one ormore subsequent frames. For example, the augmented reality system mayselectively read out the at least one subsequent frame from at least oneframe buffer. Such may take advantage of over rendering where frames areover rendered relative to the size of the image area or field of view.The system, particularly when head mounted, will in most instances bededicated to a fixed display surface having a known area and knownresolution. This is in contrast to computers and other devices which areintended to supply signals to displays of a wide variety of sizes andresolutions. Thus, the augmented reality system selectively reads intoor out of the frame buffer(s) rather than reading in or reading out theentire frame from the frame buffer. Over rendering may prevent runningthe GPU as excessively as might otherwise be required if a new framewhere rendered to create subsequent images to show those portions thatare outside of a previous image. For instance, without over rendering,the augmented reality system would need to render a new frame each timethe end user's head was moved. With over rendering, a dedicated set ofelectronics may be employed for selecting or reading out the desiredportions of the over rendered frames, essentially moving a window withina previously rendered frame.

FIG. 12 shows a method 1200 of operation in an augmented reality system,according to one illustrated embodiment. The method 1200 may be employedin executing acts 1108 and 1110 of the method 1100 of FIG. 11.

At 1202, the augmented reality system (e.g., a controller subsystemand/or processor thereof) selects at least a first set of pixels of theframe such that the first set of pixels are in a given direction (e.g.,same direction, opposite direction) with respect to a direction of adetected head movement.

At 1202, the augmented reality system adjusts a size of pixels of theselected set(s) as presented to the user of the pixels of the first setof pixels of the at least one subsequent frame.

For example, the augmented reality system may select a first set ofpixels of the frame such that the first set of pixels are positioned ina same direction as a direction of the detected head movement, relativeto other pixels. For instance, the first set of pixels may be orientedrelatively toward a left in an image relative to a second set of pixelsoriented generally to a right in the image. For instance, the first setof pixels may be oriented relatively toward a top in an image relativeto a second set of pixels oriented generally to a bottom in the image.The augmented reality system may provide one or more subsequent framesor images where the pixels of the first set have an increased size,relative to some other pixels in the subsequent frame(s). Such can atleast partially accommodate or at least partially compensate for aspreading between pixels that may result from rapid head movement whichthe augmented reality system cannot keep up with.

For example, the augmented reality system may select a first set ofpixels of the frame such that the first set of pixels are positioned inan opposite direction from a direction of the detected head movement,relative to other pixels. The augmented reality system may provide oneor more subsequent frames or images where the pixels of the first sethave a decreased size, relative to some other pixels in the subsequentframe(s). Such can at least partially accommodate or at least partiallycompensate for a spreading between pixels that may result from rapidhead movement which the augmented reality system cannot keep up with.

Adjusting (e.g., increasing, decreasing) a size of the pixels ofselected set(s) may include adjusting a variable focus element.Adjusting (e.g., increasing, decreasing) a size of the pixels ofselected set(s) may include adjusting a variable size source. Adjusting(e.g., increasing, decreasing) a size of the pixels of selected set(s)may include adjusting a jitter.

As a further example, the augmented reality system may select a firstset of pixels of the frame such that the first set of pixels arepositioned in a same direction as a direction of the detected headmovement relative to other pixels. The augmented reality system mayprovide one or more subsequent frames or images where the pixels of thefirst set have an increased intensity, relative to some other pixels inthe subsequent frame(s). Such can at least partially accommodate or atleast partially compensate for a spreading between pixels that mayresult from rapid head movement which the augmented reality systemcannot keep up with.

As yet an even further example, the augmented reality system may selecta first set of pixels of the frame such that the first set of pixels arepositioned in an opposite direction as a direction of the detected headmovement. The augmented reality system may provide one or moresubsequent frames or images where the pixels of the first set have adecreased intensity, relative to some other pixels in the subsequentframe(s). Such can at least partially accommodate or at least partiallycompensate for a spreading between pixels that may result from rapidhead movement which the augmented reality system cannot keep up with.

As noted above, the augmented reality system may adjust only the size ofthe selected pixels, only the intensity of the selected pixels, or boththe size and intensity of selected pixels. Further, the augmentedreality system may adjust intensity of some pixels, the size of otherpixels, the intensity and size of yet even other pixels, and/or notadjust either the intensity or size of yet further pixels.

The system may also be used for dynamically updating on less than wholeframe by whole frame basis, as illustrated below. FIG. 13 shows a method00 of operation in an augmented reality system, according to oneillustrated embodiment.

At 1302, the augmented reality system (e.g., a controller subsystemand/or processor thereof) renders a first complete frame to an imagebuffer. The first complete frame comprises pixel information forsequential presentation of pixels to form an image of a number ofvirtual objects. The first complete frame may take a variety of formssuitable for various display technologies. For example, a complete framemay include pixel information suitable for forming a complete rasterscan frame, which may be an interlaced raster scan frame with twofields. Each field of the interlaced raster scan may include a pluralityof lines, a first field comprising the odd lines and a second fieldcomprising the even lines. The odd and even lines may be interlaced atleast as displayed to the end user. A particularly advantageoustechnology employs spiral scan lines. The spiral scan approach mayemploy a single field per frame, for instance consisting of a singlespiral trace. Alternatively, the spiral scan approach may employ two ormore fields per frame, for instance consisting of two or more spiraltraces, presented sequentially. The spiral traces may advantageously beinterlaced or nested simply by introducing a phase shift between eachfield of the frame. Another technology employs a Lissajous scanapproach. The Lissajous scan approach may employ a single field perframe, for instance consisting of a single Lissajous trace.Alternatively, the Lissajous scan approach may employ two or more fieldsper frame, for instance consisting of two or more Lissajous traces,presented sequentially. The Lissajous traces may advantageously beinterlaced or nested simply by introducing a phase shift between eachfield of the frame.

At 1304, the augmented reality system starts a presentation of the firstcomplete frame. Such may include reading out of the frame buffer(s), forexample to drive a light source(s) and end of one or more opticalfibers. The reading out may include dynamically determining whichportions of the frame buffer to read out.

Optionally at 1306, the augmented reality system detects a head movementof an end user exceeding a nominal head movement value. Such may employany of the various approaches previously discussed.

At 1308, the augmented reality system dynamically interrupts thepresentation of the first complete frame before completion of thepresentation of the entire first complete frame. In particular, theaugmented reality system starts presentation of an update to the firstcomplete frame. At least a portion of the pixel information in theupdate to the complete frame has changed from the first complete frame.For example, in an interlaced raster scan based system, the augmentedreality system may present all or part of a first field, replacing thesecond field with an update second field. Also for example, in aninterlaced spiral scan based system, the augmented reality system maypresent all or part of a first field (e.g., a first spiral scan line ortrace), replacing the second field with an update second field (e.g., anupdated second spiral scan line or trace, different from the originalsecond spiral scan or trace). Similarly, in an interlaced Lissajous scanbased system, the augmented reality system may present all or part of afirst field (e.g., a first Lissajous scan line or trace, i.e., acomplete figure 8 cycle), replacing the second field with an updatesecond field (e.g., an updated second Lissajous scan line or trace,different from the original second spiral scan or trace). While examplesare given in terms of fields, such is not limited to entire fields. Thepresentation may be interpreted during a presentation of a field, forexample during presentation of the first or second, or even a tertiaryfield. Presentation may be interpreted during presentation of any givenline (e.g., row of a raster scan, complete cycle of a spiral or aLissajous scan).

FIG. 14 shows a method 1400 of operation in an augmented reality system,according to one illustrated embodiment. The method 1400 may be employedin executing the method 1300 of FIG. 13.

At 1402, the augmented reality system (e.g., a controller subsystemand/or processor thereof) renders an updated first complete frame. Theupdated first complete frame includes pixel information that isdifferent in at least one respect from the pixel information of thefirst complete frame.

Rendering an updated first complete frame may include rendering theupdate complete frame with a first field and at least a second field.The second field may be interlaced with the first field, typically bepresenting sequentially following presentation of the first field. Forinstance, the first field may consist of even numbered lines in a rasterscan while the second field consists of odd number lines. Also forexample, a first field may consist of a first spiral scan line or afirst Lissajous scan line, while the second field consists of a secondspiral scan line or a second Lissajous scan line. Thus, rendering anupdated first complete frame may include rendering the updated completeframe with a first field and at least a second field, the second fieldinterlaced with at least the first field.

At 1404, the augmented reality system presents a portion of the updatedfirst complete frame in lieu of a counterpart portion of the firstcomplete frame. Thus, a portion of the updated frame is substitute forall or a portion of the first complete frame following the interruptionof the original not updated first complete frame.

For example, the augmented reality system may present a second field ofthe updated first complete frame in lieu of a second field of theoriginal (i.e., not updated) first complete frame. Also for example, theaugmented reality system may present a second portion of a first fieldalong with a second field of the updated first complete frame in lieu ofa corresponding portion of a first field and the entire second field ofthe first original (i.e., not updated) first complete frame.

Also for example, the augmented reality system may present a portion(e.g., line, part of line, set of pixels, pixel) of a field of theupdated first complete frame in lieu of a counterpart portion of acounterpart field of the first complete frame. For instance, theaugmented reality system may present a portion of a field of the updatedfirst complete frame of a raster scan frame in lieu of a counterpartportion of a counterpart field of the original (i.e., not updated) firstcomplete frame of the raster scan frame.

As another example, the augmented reality system may present a line ofthe updated first complete frame in lieu of a counterpart line of theoriginal (i.e., not updated) first complete frame. As yet anotherexample, the augmented reality system may present a spiral line of theupdated first complete frame in lieu of a counter spiral line of theoriginal (i.e., not updated) first complete frame. As a further example,the augmented reality system may present a portion of a line of theupdated first complete frame in lieu of a counterpart portion of acounterpart line of the original (i.e., not updated) first completeframe. As yet a further example, the augmented reality system maypresent at least one pixel of the updated first complete frame in lieuof a counterpart at least one pixel of the original (i.e., not updated)first complete frame. As still an additional example, the augmentedreality system may present one full cycle of a Lissajous pattern scan ofthe updated first complete frame in lieu of a counter part one fullcycle of a Lissajous pattern scan of the original (i.e., not updated)first complete frame.

FIG. 15 shows a method 1500 of operation in an augmented reality system,according to one illustrated embodiment. The method 1500 may be employedin executing the method 1300 of FIG. 13.

At 1502, the augmented reality system (e.g., a controller subsystemand/or processor thereof) renders a first complete frame to a framebuffer. The first complete frame may, for example, include a first fieldand at least a second field. The first field may, for example, includepixel information for at least a first spiral scan line and the secondfield may include pixel information for at least a second spiral scanline. The scan line(s) of the second field may be interlaced with thescan line(s) of the first field. The first field may, for example,include pixel information for at least a first Lissajous scan line andthe second field may include pixel information for at least a secondLissajous scan line. The scan line(s) of the second field may beinterlaced with the scan line(s) of the first field. Interlacing of scanlines for both spiral and Lissajous scan patterns may be efficientlyachieved with phase shifts. The number of fields or scan lines can begreater than two, for example three, four, eight, sixteen or more.

At 1504, the augmented reality system starts to read out of the framebuffer, which stores the first complete frame. The augmented realitysystem may drive the light sources and a yoke or other device orstructure to generate an image based on the pixel data specified in theframe from the image buffer.

At 1506, the augmented reality system renders an updated first completeframe to the frame buffer. The updated first complete frame includespixel information that specifies a frame, a portion of which has changedfrom the information specified by the original (i.e., not updated) firstcomplete frame

At 1508, the augmented reality system starts to read out the updatedfirst complete frame before completion of the reading out of the firstcomplete frame from the frame buffer, thereby interrupting presentationof the original (i.e., not updated) first complete frame. Someimplementations may take advantage of having two or more frame buffers,allowing rendering to one frame buffer while a frame is being read outof the other frame buffer. Such should not be considered limiting in thevarious implementations of augmented reality systems may employ one,two, three or even more frame buffers.

FIG. 16 shows a method 1600 of operation in an augmented reality system,according to one illustrated embodiment. The method 1600 may be employedin executing the method 1300 of FIG. 13.

At 1602, the augmented reality system (e.g., a controller subsystemand/or processor thereof) generates pixel information for a first scanline (e.g., spiral, Lissajous).

Optionally at 1604, the augmented reality system generates pixelinformation for a second scan line (e.g., spiral, Lissajous) phaseshifted relative to the first scan line (e.g., spiral, Lissajous). Phaseshifting advantageously interfaces or nests the second scan line withthe first scan line for spiral and Lissajous scan lines.

Optionally at 1606, the augmented reality system generates pixelinformation for a third scan line (e.g., spiral, Lissajous) phaseshifted relative to the second scan line (e.g., spiral, Lissajous).Phase shifting advantageously interfaces or nests the third scan linewith the first and the second scan lines for spiral and Lissajous scanlines.

Optionally at 1608, the augmented reality system generates pixelinformation for a fourth scan line (e.g., spiral, Lissajous) phaseshifted relative to the third scan line (e.g., spiral, Lissajous). Phaseshifting advantageously interfaces or nests the fourth scan line withthe first, the second, and the third scan lines for spiral and Lissajousscan lines.

FIG. 17 shows a method 1700 of operation in an augmented reality system,according to one illustrated embodiment.

At 1702, for each of a plurality of frames the augmented reality system(e.g., a controller subsystem and/or processor thereof) determine arespective resolution for each of at least two portions of therespective frame. Portions may be fields, lines, other subdivision, oreven individual pixels.

At 1704, the augmented reality system causes a presentation of images ofvirtual objects based on the plurality of frames, at least some of theimages having a variable resolution within the image as presented to anend user. For example, spacing between adjacent pixels may differ fromone portion to another.

FIG. 18 shows a method 1800 of operation in an augmented reality system,according to one illustrated embodiment. The method 1800 may be employedin executing the method 1700 of FIG. 17.

At 1802, the augmented reality system (e.g., a controller subsystemand/or processor thereof) render frames as respective pixel data for aspiral scan pattern.

At 1804, the augmented reality system adjusts the amplitude of a drivesignal between the time of presenting a first portion of a first one ofthe frames and presenting a second portion of the first one of theframes. This change in amplitude results in the variable resolution inthe image that corresponds to the first one of the frames. The augmentedreality system may for example vary a slope or ramp of the drive signal.Such is particularly useful where used with a spiral scan pattern. Forinstance, a first field of a frame may have one slope or ramp which thesecond field has a different slope or ramp, thereby changing theeffective resolution with a single frame. The higher resolution or pixeldensity may be employed at or proximate locations of end user interestor attraction, while lower resolution or pixel density may be used awayfrom such locations. Where a center or image is shifted toward a centerof an end user's attraction or focus, the high resolution may appeararound the center of an image, while surround portions appear at lowerresolutions. Such essentially implements what can be denominated afoviated display, with steerable pixels.

FIG. 19 shows a method 1900 of operation in an augmented reality system,according to one illustrated embodiment. The method 1800 may be employedin conjunction or as part of executing the method 1700 of FIG. 17.

At 1902, the augmented reality system (e.g., a controller subsystemand/or processor thereof) assesses a point of attention in a least afirst image for the end user. The augmented reality system may use anyof the previously described techniques for assessing such. For example,determining whether and where a new virtual object will appear, or wherea virtual object will move to within a field of view of the end user.Also for example, the augmented reality system may assess a relativeattractiveness of the virtual objects (e.g., speed, color, size,brightness, shimmer). Such may also employ eye tracking information,indicative of a location in a field of view which the eyes of the enduser are tracking or focused on.

Eye tracking information may, for example, be supplied via one or morehead worn transducers, for instance head worn cameras. Such eye trackinginformation may, for example, be discerned by projecting light at theend user's eyes, and detecting the return or reflection of at least someof that projected light. For instance, a projection subsystem thatcreates or projects the images may project a pixel, dot or other elementof light from the at least one optical fiber to create a glint off theend user's cornea. Eye tracking may employ, one, two, three or even morespots or dots of light. The more spots or dots of light, the moreinformation may be discerned. The light sources (e.g., laser diodes) maybe pulsed or modulated, for example in synchronization with a frame rateof a camera or image sensor. In which case, spots or dots may appear aslines as eyes move. A direction of the lines as the lines trace across asensor may indicate a direction of the eye movement. Orientation of thelines (e.g., vertical, horizontal, diagonal) indicates orientation ofthe eye movement. A length of a line is indicative of speed of eyemovement.

For eye tacking, the light may be modulated (e.g., temporally,intensity) to increase the signal to noise ratio. Additionally oralternatively, the light may be of a specific wavelength (e.g.,near-IR), allowing such to be distinguished from background light oreven the light that forms the image that the end user is watching. Thelight may be modulated to reduce a total amount of energy (e.g., heat)that the augmented reality system is providing to the eye(s). The glintmay be returned via the same, or another, optical fiber to a sensor. Thesensor may, for example take the form of a two-dimensional image sensor,for instance a CCD sensor or a CMOS sensor.

Thus, the augmented reality system may detect and track relativemovements of the eyes, providing an indication of a point or location ofthe end user's attention or focus. The augmented reality system maylogically associate a virtual object or virtual event (e.g., appearanceor movement of a virtual object) with the identified point or locationof the end user's attention or focus. For example, the augmented realitysystem may designate a virtual object that appears at or at leastproximate the point or location of the end user's attention or focus asan attractive virtual object to the end user.

At 1904, the augmented reality system adjusts (e.g., increases,decreases) resolution in at least one portion of at least one subsequentimage. The augmented reality system may employ any of the varioustechniques described herein, as well as other techniques to adjustresolution of a portion of a subsequent page relative to other portionsof the same subsequent page.

FIG. 20 shows a method 2000 of operation in an augmented reality system,according to one illustrated embodiment. The method 2000 may be employedin executing act 1904 of the method 1900 of FIG. 19.

At 2002, the augmented reality system (e.g., a controller subsystemand/or processor thereof) increases a resolution in a portion of the atleast one subsequent image that is at least proximate the assessed pointof attention relative to other portions of the at least one subsequentimage. As previously explained, resolution may be adjusted for spiralscan patterns by controlling a magnitude or amplitude (e.g., current,voltage) of the drive signal. The resolution may be adjusted byadjusting a slope of the drive signal. Resolution may thus be increasedby increasing the amplitude of the drive signal, while the phase is leftunchanged.

At 2004, the augmented reality system decreases a resolution in aportion of the at least one subsequent image that is distal to theassessed point of attention relative to other portions of the at leastone subsequent image. Resolution may be decreased by decreasing theamplitude of the drive signal, while the phase is left unchanged.

In some implementations the resolution is only increased, increasing insome portions while neither increasing nor decreasing in other portions.In other implementations the resolution is only decreased, decreasing insome portions while neither increasing nor decreasing in other portions.In yet still other implementations resolution is both increased in someportions while decreased in other portions.

FIG. 21 shows a method 2100 of operation in an augmented reality system,according to one illustrated embodiment. The method 2100 may be employedin conjunction with the method 1700 of FIG. 17.

At 2102, the augmented reality system (e.g., a controller subsystemand/or processor thereof) processes eye tracking data. The eye trackingdata is indicative of at least an orientation of at least one eye of theend user. The eye tracking data is supplied via at least one transducer.For example, the eye tracking data may be supplied via a head worntransducer. In some implementations, eye tracking data is collected viaan optical fiber, with a transducer positioned at or proximate a distalend thereof. For instance, the optical fiber may collect light reflectedfrom a portion of the end user's eye(s), which may be a glint. Theoptical fiber may be the same optical fiber as used to create the imagefor display or projection to the end user.

At 2104, the augmented reality system processes head tracking data. Thehead tracking data is indicative of at least an orientation of a head ofthe end user. The head tracking data may be supplied via at least onetransducer.

For example, one or more head worn or mounted transducers such asinertial sensors (e.g., gyroscopic sensors, accelerometers). Headmovement tracking may be implemented using one or more head worn or headmounted light sources and at least one sensor. Head tracking may employ,one, two, three or even more spots or dots of light. The more spots ordots of light, the more information may be discerned. The light sources(e.g., laser diodes) may be pulsed or modulated, for example insynchronization with a frame rate of a camera or image sensor (e.g.,front facing camera). The laser source(s) may be modulated at afrequency that is lower than the frame rate of the camera or imagesensor. In which case, spots or dots may appear as lines as the headmove. A direction of the lines as the lines trace across a sensor mayindicate a direction of the head movement. Orientation of the lines(e.g., vertical, horizontal, diagonal) indicates orientation of the headmovement. A length of a line is indicative of speed of head movement.Reflected light may also provide information regarding objects in theambient environment, such as distance and/or geometry (e.g., planar,curved) and/or orientation (e.g., angled or perpendicular). Forinstance, one laser bean may produce information regarding direction andvelocity (e.g., length of dash or line). A second laser beam may addinformation regarding depth or distance (e.g., Z-axis). A third laserbeam may add information about the geometry and/or orientation of asurface in the ambient environment. Lasers or other light sources may bepulsed during head movement or during part of the head movement.

Additionally or alternatively, the head tracking data may be suppliedvia transducers that are not head worn. For example, a camera or imagersystem may image the end user, including the head of the end user,tracking movement thereof. Such may, for example, track movementrelative to some external reference frame, for instance a referenceframe defined by the tracking system or a room in which the trackingsystem is located.

At 2106, the augmented reality system determines a location ofappearance of a virtual object in a field of view of the end userrelative to an end user frame of reference. The appearance may be anappearance of a new virtual object, when newly introduced in a field ofview of the end user. The appearance may be an appearance of a virtualobject in a new position in an image relative to a position of thevirtual object in at least one previous image. The augmented realitysystem may employ any of the numerous techniques described elsewhereherein to determine the location of appearance of the virtual object.

The system may also use blanking to improve end user perception.

FIG. 22 shows a method 2200 of operation in an augmented reality system,according to one illustrated embodiment. The method 2200 effectivelyemploys blanking to improve end user perception experience.

At 2202, the augmented reality system (e.g., a controller subsystemand/or processor thereof) displays at least one virtual object to an enduser. The augmented reality system may render frames to a frame buffer,read out the frames to drive one or more light sources and/or yoke orother system to produce an at least bi-axial movement or trace of thelight.

At 2204, the augmented reality system detects and/or predicts anoccurrence of a head movement of the end user. The augmented realitysystem may employ any of the numerous techniques described elsewhereherein to detect and/or predict an occurrence of a head movement.Without limitation, those techniques include directly sensing headmovement, for example via inertial transducers or sensor, or via imagesfrom a head worn imager or an environmental imager that images an areain which the end user is present and visible. Those techniques alsoinclude indirectly predicting head movement, for instance by determiningwhere a new virtual object will appear, where an existing virtual objectwill move, or where particularly attractive virtual objects arepositioned in the image.

At 2206, the augmented reality system assesses whether the detectedand/or the predicted head movement exceeds or is predicted to exceed anominal head movement value. The augmented reality system may employ anyof the numerous techniques described elsewhere herein to assesseswhether the detected and/or the predicted head movement exceeds or ispredicted to exceed a nominal head movement value. Such may includesimple comparison of detected or predicted speed to a nominal speed.Such may include simple comparison of detected or predicted accelerationto a nominal acceleration. Such may include simple comparison ofdetected or predicted range to a nominal range. Such may include morecomplicated comparisons, including averages or integrations of speed,acceleration or range over multiple times during a movement. Such mayeven employ historical attributes or other information.

At 2208, the augmented reality system temporarily blanks at least aportion of the display of the at least one virtual object to the enduser. For example, the augmented reality system may stop reading fromthe frame buffer. Additionally or alternatively, the augmented realitysystem may turn off an illumination or light source. Such may includetemporarily turning off a back light of an LCD display.

FIG. 23 shows a method 2300 of operation in an augmented reality system,according to one illustrated embodiment. The method 2300 may be employedin performing the method 2200 of FIG. 22.

At 2302, the augmented reality system (e.g., a controller subsystemand/or processor thereof) processes head tracking data. The headtracking data is indicative of at least an orientation of a head of theend user. The head tracking data may be supplied via at least onetransducer, which may or may not be head worn by the end user. Theaugmented reality system may employ any of the numerous techniquesdescribed elsewhere herein to process the head tracking data.

At 2304, for each of at least some of the images presented to the enduser, the augmented reality system (e.g., a controller subsystem and/orprocessor thereof) determines a location of appearance of a virtualobject in a field of view of the end user relative to an end user frameof reference. The determining a location of appearance of a virtualobject when newly introduced in a field of view of the end user. Thedetermining a location of appearance of a virtual object in a newposition in an image relative to a position of the virtual object in atleast one previous image. The augmented reality system may employ any ofthe numerous techniques described elsewhere herein to determine alocation of appearance of a virtual object.

At 2306, the augmented reality system assesses whether determinedappearance of virtual object is sufficiently attractive. For example,the augmented reality system may assess a relative visual attractivenessof the virtual objects (e.g., speed, color, size, brightness, shimmer,transparency, special optical effects). Also for example, the augmentedreality system may assess the relative interest attractiveness (e.g.,newness, recentness, previous attention, previous identification by theend user, previous interaction with by the end user).

At 2308, the augmented reality system assesses whether the determinedlocation requires the end user to turn the end user's head relative to acurrent position of the end user's head. The augmented reality systemmay employ a current position and/or orientation of the end user's headand a relative position and/or orientation of the virtual object. Theaugmented reality system may determine a distance, for example angulardistance between a current focus of the end user and the position and/ororientation of the virtual object. The augmented reality system maydetermine whether the determined distance is within a range of eyemovement, or whether the end user must also turn their head. If the enduser must turn their head, the system may assess how far the end usermust turn their head. For example, the augmented reality system mayemploy information that specifies a relationship between eye movementand head movement for the end user. Such may indicate to what extent theend user will shift their gaze via eye movements alone, before turningtheir heads. Notably, the relationship between eye movement and headmovement may be specified for various different directions, for instancea) up to down, b) down to up, c) left to right, d) right to left, e)diagonally from lower left to upper right, f) diagonally from lowerright to upper left, g) diagonally from upper left to lower right, or h)diagonally from upper right to lower left.

At 2310, the augmented reality system predicting the occurrence of thehead movement based on the assessment. The augmented reality system mayuse one or more factors form the assessment in predicting whether headmovement will occur, a direction and/or orientation of the headmovement, and/or speed or acceleration of the head movement. Theaugmented reality system may employ historical data either end userspecific or more generic to a group of end users. The augmented realitysystem may implement one or more machine learning algorithms to increasethe accuracy of head movement prediction.

FIG. 24 shows a method 2400 of operation in an augmented reality system,according to one illustrated embodiment. The method 2400 may be employedin performing act 2208 of the method 2200 of FIG. 22.

At 2402, the augmented reality system (e.g., a controller subsystemand/or processor thereof) strobes or flashes a display or a backlight ofa display. The strobing or flashing occur over all or a portion of thedetected head movement or predicted head movement. Such mayadvantageously effectively reduce the perception of inconsistencies in aframe or presentation of virtual objects. Such may also effectivelyincrease perceived frame rate.

FIG. 25 shows a method 2500 of operation in an augmented reality system,according to one illustrated embodiment.

At 2502, the augmented reality system (e.g., a controller subsystemand/or processor thereof) detects and/or predicts an occurrence of ahead movement of an end user. For example, the augmented reality systemmay process head tracking data indicative of at least an orientation ofa head of the end user. Additionally or alternatively, the augmentedreality system may determining a location of appearance of a virtualobject in a field of view of the end user relative to an end user frameof reference, assess whether the determined location requires the enduser to turn a head of the end user, and predict an occurrence of thehead movement based on the assessment. The augmented reality system mayemploy any of the numerous techniques described elsewhere herein todetect and/or predict an occurrence of a head movement.

At 2504, the augmented reality system determines whether the detected orthe predicted head movement exceeds a nominal head movement value. Theaugmented reality system may employ any of the numerous techniquesdescribed elsewhere herein to determine whether the detected or thepredicted head movement exceeds a nominal head movement value.

At 2506, in response to determining that the detected or the predictinghead movement exceeding the nominal head movement value, the augmentedreality system selectively activates an actuator to move the projectorin at least one degree of freedom. Moving the projector may includetranslating the first optical fiber along at least one axis. Moving theprojector may include pivoting the first optical fiber about at leastone axis.

FIG. 26 shows a method 2600 of operation in an augmented reality system,according to one illustrated embodiment.

The augmented reality system may over render frames, producing framesthat are much larger than need for a maximum area and maximum resolutionof a given display technology. For example, in a head worn or mountedaugmented reality system the area available for display or projectionmay be set by the various parameters of the equipment. Likewise, whilethe augmented reality system may be capable of operating at multipledistinct resolutions, the equipment will set an upper end or maximumresolution. An over rendered frame includes pixel information for a setof pixels that exceeds the maximum area of display at the maximumresolution. Such may advantageously allow the augmented reality systemto read out only portions of frames (e.g., a portion of each field ofthe frame, if not interrupted). Such may allow the augmented realitysystem to shift an image presented to the user.

At 2602, the augmented reality system (e.g., a controller subsystemand/or processor thereof) over renders each of a plurality of frames fora defined field of view. Such requires generating more pixel informationthat would otherwise be required for the maximum area at maximumresolution. For example, the area of the frame may be increase by apercentage of the maximum area, for example increasing the pixelinformation in either a horizontal, vertical or diagonal directiondefined by the frame. The larger the frame size the more freedom theaugmented reality system will have to shift the boundaries of the imagespresented to the user.

At 2604, the augmented reality system successively buffers the overrendered frames in at least one frame buffer. The augmented realitysystem may employ a frame buffer that is larger than the frame sizerequired for the maximum display size at maximum resolution. Someimplementations employ multiple frame buffers. Such may facilitateinterruption of presentation for frames, as described elsewhere herein.

At 2606, the augmented reality system determines a portion of arespective image to present. The augmented reality system may determinethe portion based on any of a variety of factors. For example, thefactors may be indicative of a location in an image or scene at whichthe end user is paying attention to, focusing on or has otherwiseattracted the attention of the end user. Again various techniques may beemployed, including but not limited to eye tracking. Also for example,the factors may be indicative of a location in an image or scene atwhich the end user is predicted to pay attention to, focus on or willotherwise attract the attention of the end user. Again varioustechniques may be employed including but not limited to identifyingnewly appearing virtual objects, fast or rapidly moving virtual objects,virtual objects that are visually attractive, virtual objects that havebeen previously designated (e.g., designated by the end user or bypreviously tracking the end user's interaction) and/or virtual objectsthat are attractive of attention based on the inherent nature of thevertical object. Virtual objects that are attractive of attention basedon the inherent nature of the vertical object may, for example, includea virtual object that visually represents an object or item of concernor suspense to either generalized end users or a specific end user(e.g., imminent threats).

At 2608, the augmented reality system selectively reads out of theportion of the over rendered frame from the frame buffer. The portion isbased at least in part on the determined portion of the respective imageto present. For example, the portion that is read out may have a centerthat is shifted to be proximate, or even match or co-align, with theidentified location. The identified location may, for example, be alocation in a previous image or frame that has attracted an end user'sattention. The identified location may, for example, be a location in asubsequent frame which the augmented reality system has predicted willattract the end user's attention.

FIG. 27 shows a method 2700 of operation in an augmented reality system,according to one illustrated embodiment. The method 2700 may be employedin performing the method 2600 of FIG. 26. For instance, the method 2700may be used to predict a location in a subsequent frame or image whichwill attract the end user's attention.

At 2702, for each of at least some of the frames the augmented realitysystem (e.g., a controller subsystem and/or processor thereof)determines a location of appearance of a virtual object in a field ofview of the end user relative to an end user frame of reference.

At 2704, the augmented reality system selectively reads out of the framebuffer based at least in part on the determining a location ofappearance of a virtual object in a field of view. For example, theportion that is read out may have a center that is shifted to beproximate, or even match or co-align, with the identified location.Alternative, the boundaries of the portion that is read out may beshifted to encompass the determined location an immediately surroundingareas, in two- or even three-dimensions. For example, the augmentedreality system may select a portion (e.g., 80%) of the entire overrendered frame to be read out of the frame buffer for presentation tothe end user. The augmented reality system may select that portion suchthat the boundaries are shifted relative to a current location of theend user's attention, for instance in images currently being presentedto the end user. The augmented reality system may select the boundariesbased on a combination of the current location and the predictedlocation, while setting the boundary such that both locations will bepresented to the end user in the subsequently presented image(s).

FIG. 28 shows a method 2800 of operation in an augmented reality system,according to one illustrated embodiment. The method 2800 may be employedin performing the method 2600 of FIG. 26. For example, the method 2800may be employed to determine a location in an image that has attractedor is predicted to attract the attention of the end user.

At 2802, the augmented reality system (e.g., a controller subsystemand/or processor thereof) determines the location of appearance of a newvirtual object, when the new virtual is newly introduced in the field ofview of the end user. The augmented reality system may employ any of thevarious techniques described herein to identify the introduction of avirtual object, which is new relative to immediately previous frames orimages presented to the end user. Thus, even if a virtual object haspreviously been presented to the end user at some other portion of apresentation, the virtual object may be identified as newly introducedif a sufficient number of intervening images have been presented to makethe reintroduction of the virtual object attract the attention of theend user.

At 2804, the augmented reality system determines the location ofappearance of the virtual object in a new position in the frame relativeto a position in at least one previous frame. The augmented realitysystem may employ any of the various techniques described herein toidentify the moving of a virtual object to a new or different positionin a number of images, which is move relative to immediately previousframes or images presented to the end user. Thus, even if a virtualobject has previously been presented to the end user at that somelocation in some other portion of a presentation, the virtual object maybe identified as moved or moving if a sufficient number of interveningimages have been presented to make the reappearance of the virtualobject at the previous location attract the attention of the end user.

At 2806, the augmented reality system determines a location of a virtualobject having at least a defined minimum speed in a field of view of theend user. The augmented reality system may employ any of the varioustechniques described herein to determine speed of movement of a virtualobject from image to image and to compare that speed to a defined ornominal speed. The determined speed may be relative to a fixed referenceframe in the image or relative to other virtual objects and/or physicalobjects that appear in the image.

At 2808, the augmented reality system determines a portion of arespective image to present based at least in part on a location of thevirtual object in an image. The augmented reality system may employ anyof the various techniques described herein to determine the portion of arespective image to present. The determination may be based on any of avariety of factors. The factors may, for example, include factors ordata that is indicative of a location in an image or scene at which theend user is paying attention to, focusing on or has otherwise attractedthe attention of the end user. The factors may, for example, includefactors or data that is indicative of a location in an image or scene atwhich the end user is predicted to pay attention to, focus on or willotherwise attract the attention of the end user. The augmented realitysystem may employ any of the various techniques described elsewhereherein for identifying locations that have attracted the end user'sattention, whether via actual detection or via prediction.

At 2810, the augmented reality system reads out a portion of the framebuffer for at least one subsequent frame. The portion that is read outshifts a center of the image at least toward the determined portion ofthe respective image that will be presented. The augmented realitysystem may employ any of the various techniques described herein to readout of the frame buffer a portion of a frame that shifts the center orboundaries of the image based on the location of an end user's actual orpredicted center of attention.

FIG. 29 shows a method 2900 of operation in an augmented reality system,according to one illustrated embodiment. The method 2900 may be employedin performing the method 2600 of FIG. 26. In particular, the method 2900may be employed to determine a which portion of a frame to read outbased on a predicted head movement of the end user.

At 2902, the augmented reality system (e.g., a controller subsystemand/or processor thereof) predicts an occurrence of a head movement ofthe end user. The augmented reality system may employ any of the varioustechniques described herein to predict head movement. Such techniquesinclude, but are not limited to, detecting the appearance in images of anew virtual object, a moving virtual object, a virtual objecting movingrapidly, a previously selected virtual object and/or a visuallyattractive virtual object.

At 2904, the augmented reality system determines a portion of arespective frame or image to present based at least in part on thepredicted head movement. The augmented reality system may employ any ofthe various techniques described herein to determine the portion of theframe to be used. For example, the augmented reality system may selectthe portion such that the boundaries encompass the location of apredicted end point of the predicted head movement. Where the headmovement prediction is predicated on the appearance of a virtual object(e.g., newly introduced, moving, attractive appearance, previouslyselected by end user), the end point may be coincident with the locationof that virtual object in a subsequent frame or image.

FIG. 30 shows a method 3000 of operation in an augmented reality system,according to one illustrated embodiment.

The augmented reality system may over render frames, producing framesthat are much larger than need for a maximum area and maximum resolutionof a given display technology. For example, in a head worn or mountedaugmented reality system the area available for display or projectionmay be set by the various parameters of the equipment. Likewise, whilethe augmented reality system may be capable of operating at multipledistinct resolutions, the equipment will set an upper end or maximumresolution. An over rendered frame includes pixel information for a setof pixels that exceeds the maximum area of display at the maximumresolution. Such may advantageously allow the augmented reality systemto read out only portions of frames (e.g., a portion of each field ofthe frame, if not interrupted). Such may allow the augmented realitysystem to shift an image presented to the user.

At 3002, the augmented reality system (e.g., a controller subsystemand/or processor thereof) over renders each of a plurality of frames fora defined field of view. Such requires generating more pixel informationthat would otherwise be required for the maximum area at maximumresolution. For example, the area of the frame may be increase by apercentage of the maximum area, for example increasing the pixelinformation in either a horizontal, vertical or diagonal directiondefined by the frame. The larger the frame size the more freedom theaugmented reality system will have to shift the boundaries of the imagespresented to the user.

At 3004, the augmented reality system determines a portion of arespective image to present. The augmented reality system may determinethe portion based on any of a variety of factors. For example, thefactors may be indicative of a location in an image or scene at whichthe end user is paying attention to, focusing on or has otherwiseattracted the attention of the end user. Again various techniques may beemployed, including but not limited to eye tracking. Also for example,the factors may be indicative of a location in an image or scene atwhich the end user is predicted to pay attention to, focus on or willotherwise attract the attention of the end user. Again varioustechniques may be employed including but not limited to identifyingnewly appearing virtual objects, fast or rapidly moving virtual objects,virtual objects that are visually attractive, virtual objects that havebeen previously designated (e.g., designated by the end user or bypreviously tracking the end user's interaction) and/or virtual objectsthat are attractive of attention based on the inherent nature of thevertical object. Virtual objects that are attractive of attention basedon the inherent nature of the vertical object may, for example, includea virtual object that visually represents an object or item of concernor suspense to either generalized end users or a specific end user(e.g., imminent threats).

At 3006, the augmented reality system dynamically addresses one or moredetermined portions of the over rendered frame into the buffer(s). Thedetermined portion(s) may, for example, have a center that is shifted tobe proximate, or even match or co-align, with an identified location ofan end user's attraction, interest or focus. The identified locationmay, for example, be a location in a previous image or frame that hasattracted an end user's attention. The identified location may, forexample, be a location in a subsequent frame which the augmented realitysystem has predicted will attract the end user's attention. Someimplementations employ multiple frame buffers. Such may facilitateinterruption of presentation for frames, as described elsewhere herein.

At 3008, the augmented reality system reads out of the determinedportion(s) of the over rendered frame from the frame buffer.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

What is claimed is:
 1. A user display device, comprising: a housingframe mountable on a head of a user; a lens mountable on the housingframe; and a projection subsystem coupled to the housing frame todetermine a location of appearance of a display object in a field ofview of the user based at least in part on at least one of a detectionof a head movement of the user and a prediction of a head movement ofthe user, and to project the display object to the user based on thedetermined location of appearance of the display object.
 2. The userdisplay device of claim 1, wherein the location of appearance of thedisplay object is moved in response to the at least one of the detectionof the head movement of the user or prediction of the head movement ofthe user that exceeds or it predicted to exceed a nominal head movementvalue.
 3. The user display device of claim 1, wherein the prediction ofthe head movement of the user is based on a prediction of a user's shiftin focus.
 4. The user display device of claim 1, wherein the predictionof the head movement of the user is based on a set historical attributesof the user.
 5. The user display device of claim 1, further comprising afirst pair of cameras mountable on the housing frame to track a movementof the user's eyes and estimate a depth of focus of the user's eyesbased on the tracked eye movements.
 6. The user display device of claim5, wherein the projection subsystem projects the display object based onthe estimated depth of focus.
 7. The user display device of claim 1,further comprising a second pair of cameras mountable on the housingframe to capture a field of view image as seen by the user's eyes,wherein the field of view image contains at least one physical object.8. The user display device of claim 7, wherein the projection sub systemprojects the display object in a manner such that the display object andthe physical object captured through the second pair of cameras areinter-mixed and appear together in the same frame.
 9. The user displaydevice of claim 8, wherein the location of appearance is based at leastin part on the physical object.
 10. The user display device of claim 9,wherein the display object and the physical object have a predeterminedrelationship.
 11. The user display device of claim 5, wherein thecaptured field of view image is used to gather information regardingmovements of the head of the user, wherein the information regardingmovements of the head of the user comprises a center of attention of theuser, an orientation of the head of the user, a direction of the head ofthe user, a speed of movement of the head of the user, an accelerationof the head of the user and a distance of the head of the user inrelation to a local environment of the user.
 12. The user display deviceof claim 1, wherein the lens comprises at least one transparent surfaceto selectively allow a transmission light such that the user is able toview a local environment.
 13. The user display device of claim 12,wherein the projection subsystem projects the display object in a mannersuch that the user views both the display object and the localenvironment as viewed through the transparent surface of the lens. 14.The user display device of claim 1, further comprising at least oneintertial transducer to capture a set of intertial measurementsindicative of movement of the head of the user, wherein the set ofintertial measurements comprises a speed of movement of the head of theuser, an acceleration of movement of the head of the user, a directionof movement of the head of the user, a position of the head of the userand an orientation of the head of the user.
 15. The user display deviceof claim 1, further comprising at least one light source to illuminateat least one of the head of the user and a local environment of theuser.
 16. The user display device of claim 1, wherein the projection subsystem adjusts at least one of a perceived size, an intensity and aresolution of a set of pixels associated with the display object tocompensate for the at least one of the detected head movement and thepredicted head movement.
 17. The user display device of claim 1, whereinthe display object is one of a virtual object and an augmented virtualobject.